Methods and apparatuses for processing video signal

ABSTRACT

Embodiments of the present invention provide video signal processing methods. A video signal encoding method according to an embodiment of the present invention comprises checking a transform block including residual samples except a prediction sample from a picture of the video signal, generating transform coefficients through a transform for the residual samples of the transform block based on a size of the transform block, and performing quantization and entropy coding the transform coefficients, wherein generating the transform coefficients includes, applying a forward primary transform to each of a horizontal direction and vertical direction of the transform block including the residual samples, and not applying a forward non-separable secondary transform to the transform block to which the primary transform has been applied when the size of the transform block is smaller than or equal to 4×4, and applying the forward non-separable secondary transform to the transform block when the size of the transform block is greater than 4×4.

TECHNICAL FIELD

The present disclosure relates to a method and apparatus for processinga video signal and, more particularly, to a method and apparatus forencoding and decoding a video signal by omitting the application of anon-separable secondary transform to a transform block having a 4×4size.

BACKGROUND ART

Compression encoding means a series of signal processing techniques fortransmitting digitized information through a communication line ortechniques for storing the information in the form that is suitable fora storage medium. The media including a video, an image, an audio, andthe like may be the target for the compression encoding, andparticularly, the technique of performing the compression encodingtargeted to the video is referred to as a video image compression.

Next generation video content is supposed to have the characteristics ofhigh spatial resolution, high frame rate and high dimensionality ofscene representation. In order to process such contents, drasticincrease of memory storage, memory access rate and processing power willbe resulted.

Accordingly, a coding tool for processing more efficiently the nextgeneration video contents needs to be designed. In particular, videocodec standard after high efficiency video coding (HEVC) standardrequires an efficient transform technology for transforming a videosignal of a spatial domain into a frequency domain.

DISCLOSURE Technical Problem

For an implementation of an efficient transform technology, there is aneed for a method and apparatus for providing a low-complexity transformtechnology.

Accordingly, embodiments of the present disclosure provide a videosignal processing method and apparatus for performing a transform withlow complexity.

Furthermore, embodiments of the present disclosure provide a videosignal processing method and apparatus capable of reducing computationalcomplexity through a non-separable secondary transform.

The technical objects to be achieved by the present disclosure are notlimited to those that have been described hereinabove merely by way ofexample, and other technical objects that are not mentioned can beclearly understood from the following descriptions by those skilled inthe art to which the present disclosure pertains.

Technical Solution

A method of encoding a video signal according to an embodiment of thepresent disclosure includes checking a transform block includingresidual samples except a prediction sample from a picture of the videosignal, generating transform coefficients through a transform for theresidual samples of the transform block based on a size of the transformblock, and performing quantization and entropy coding the transformcoefficients. Generating the transform coefficients may include applyinga forward primary transform to each of a horizontal direction andvertical direction of the transform block including the residualsamples, and omitting the application of a forward non-separablesecondary transform to the transform block to which the primarytransform has been applied when the size of the transform block issmaller than or equal to 4×4 and applying the forward non-separablesecondary transform to the transform block when the size of thetransform block is greater than 4×4.

According to an embodiment of the present disclosure, applying theforward non-separable secondary transform to the transform block mayinclude applying the forward non-separable secondary transform to aleft-top 4×4 region of the transform block.

Furthermore, the forward non-separable secondary transform according toan embodiment of the present disclosure may include at least one of areduced secondary transform (RST) having an L×16 matrix, wherein L issmaller than or equal to 16, a sparse orthonormal transform (SOT) havinga matrix of a 16×16 form, a Givens rotation-based transform having 16inputs and 16 outputs, or a layered givens transform having 16 inputsand 16 outputs and applying a designated permutation between twoneighboring Givens rotation layers.

Furthermore, the forward primary transform may include a horizontaltransform and a vertical transform for the transform block, and each ofthe horizontal transform and the vertical transform may be configured bya combination of DCT-2, DST-7 or DCT-8.

A method of decoding a video signal according to an embodiment of thepresent disclosure includes generating a transform block includingtransform coefficients by performing entropy decoding and inversequantization on the video signal, generating residual signals through aninverse transform for the transform coefficients of the transform blockbased on a size of the transform block, and generating a reconstructedpicture by generating a prediction signal through prediction from theresidual signals. Performing the inverse quantization may includeomitting the application of an inverse non-separable secondary transformto the transform block when the size of the transform block is smallerthan or equal to 4×4 and applying the inverse non-separable secondarytransform to the transform block when the size of the transform block isgreater than 4×4, and applying an inverse primary transform to thetransform block separately in a horizontal direction and a verticaldirection.

According to an embodiment of the present disclosure, applying theinverse non-separable secondary transform to the transform block mayinclude applying the inverse non-separable secondary transform to aleft-top 4×4 region of the transform block.

The inverse non-separable secondary transform according to an embodimentof the present disclosure includes at least one of a reduced secondarytransform (RST) having a matrix of an L×16 form, wherein L is smallerthan or equal to 16, a sparse orthonormal transform (SOT) having amatrix of a 16×16 form, a Givens rotation-based transform having 16inputs and 16 outputs, or a layered givens transform having 16 inputsand 16 outputs and applying a designated permutation between twoneighboring Givens rotation layers. The inverse primary transform mayinclude a horizontal transform and a vertical transform for thetransform block, and the horizontal transform and the vertical transformmay be configured by a combination of DCT-2, DST-7, and DCT-8.

An apparatus for encoding a video signal according to an embodiment ofthe present disclosure may include a memory storing the video signal andan encoder functionally coupled to the memory and configured toencoding-process the video signal. The encoder may be configured tocheck a transform block including residual samples except a predictionsample from a picture of the video signal, generate transformcoefficients through a transform for the residual samples of thetransform block based on a size of the transform block, and performquantization and entropy coding the transform coefficients Whengenerating the transform coefficients through the transform for theresidual samples of the transform block, the encoder may be configuredto apply a forward primary transform to each of a horizontal directionand vertical direction of the transform block including the residualsamples, omit the application of a forward non-separable secondarytransform to the transform block to which the primary transform has beenapplied when the size of the transform block is smaller than or equal to4×4, and apply the forward non-separable secondary transform to thetransform block when the size of the transform block is greater than4×4.

The encoder according to an embodiment of the present disclosure may beconfigured to apply the forward non-separable secondary transform to aleft-top 4×4 region of the transform block when the size of thetransform block is greater than 4×4.

The forward non-separable secondary transform according to an embodimentof the present disclosure may include at least one of a reducedsecondary transform (RST) having an L×16 matrix, wherein L is smallerthan or equal to 16, a sparse orthonormal transform (SOT) having amatrix of a 16×16 form, a Givens rotation-based transform having 16inputs and 16 outputs, or a layered givens transform having 16 inputsand 16 outputs and applying a designated permutation between twoneighboring Givens rotation layers.

The forward primary transform according to an embodiment of the presentdisclosure may include a horizontal transform and a vertical transformfor the transform block, and each of the horizontal transform and thevertical transform may be configured by a combination of DCT-2, DST-7 orDCT-8.

An apparatus for decoding a video signal according to an embodiment ofthe present disclosure may include a memory storing the video signal anda decoder functionally coupled to the memory and configured todecoding-process the video signal. The decoder may be configured togenerate a transform block including transform coefficients byperforming entropy decoding and inverse quantization on the videosignal, generate residual signals through an inverse transform for thetransform coefficients of the transform block based on a size of thetransform block, and generate a reconstructed picture by generating aprediction signal through prediction from the residual signals. Whengenerating the residual signals through the inverse transform for thetransform coefficients of the transform block, the decoder may beconfigured to omit the application of an inverse non-separable secondarytransform to the transform block when the size of the transform block issmaller than or equal to 4×4, apply the inverse non-separable secondarytransform to the transform block when the size of the transform block isgreater than 4×4, and apply an inverse primary transform to thetransform block separately in a horizontal direction and a verticaldirection.

The decoder according to an embodiment of the present disclosure may beconfigured to apply the inverse non-separable secondary transform to aleft-top 4×4 region of the transform block when the size of thetransform block is greater than 4×4.

The inverse non-separable secondary transform according to an embodimentof the present disclosure includes at least one of a reduced secondarytransform (RST) having a matrix of an L×16 form, wherein L is smallerthan or equal to 16, a sparse orthonormal transform (SOT) having amatrix of a 16×16 form, a Givens rotation-based transform having 16inputs and 16 outputs, or a layered givens transform having 16 inputsand 16 outputs and applying a designated permutation between twoneighboring Givens rotation layers. The inverse primary transform mayinclude a horizontal transform and a vertical transform for thetransform block, and the horizontal transform and the vertical transformmay be configured by a combination of DCT-2, DST-7, and DCT-8.

Advantageous Effects

According to an embodiment of the present disclosure, a transform matrixcan be designed with low complexity.

Furthermore, according to an embodiment of the present disclosure, acomputational load can be reduced by selectively applying anon-separable secondary transform based on the size of a transformblock.

Effects that could be achieved with the present disclosure are notlimited to those that have been described hereinabove merely by way ofexample, and other effects and advantages of the present disclosure willbe more clearly understood from the following description by a personskilled in the art to which the present disclosure pertains.

DESCRIPTION OF DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the present disclosure and constitute a part of thedetailed description, illustrate embodiments of the present disclosureand serve to explain technical features of the present disclosuretogether with the description.

FIG. 1 illustrates a schematic block diagram of an encoder performingencoding of a video signal as an embodiment to which the presentdisclosure is applied.

FIG. 2 illustrates a schematic block diagram of a decoder performingdecoding of a video signal as an embodiment to which the presentdisclosure is applied.

FIG. 3 illustrates embodiments to which the present disclosure may beapplied, wherein FIGS. 3A, 3B, 3C, and 3D are diagrams for describingblock division structures based on a quadtree, a binary tree, a ternarytree, and an asymmetric tree, respectively.

FIGS. 4 and 5 are embodiments to which the present disclosure isapplied, wherein FIG. 4 illustrates a schematic block diagram oftransform and quantization units and inverse quantization unit andinverse transform units within the encoder, and FIG. 5 illustrates aschematic block diagram of inverse quantization and inverse transformunits within the decoder.

FIGS. 6a and 6b illustrate examples of tables for determining atransform type in a horizontal direction and vertical direction for eachprediction mode.

FIG. 7 is an embodiment to which the present disclosure is applied, andis a flowchart illustrating an encoding process performed by multipletransform selections (MTS).

FIG. 8 is an embodiment to which the present disclosure is applied, andis a flowchart illustrating a decoding process in which MTS isperformed.

FIG. 9 is an embodiment to which the present disclosure is applied, andis a flowchart for describing a process of encoding an MTS flag and anMTS index.

FIG. 10 is an embodiment to which the present disclosure is applied, andis a flowchart for illustrating a decoding process of applying ahorizontal transform or a vertical transform to a row or column based onan MTS flag and an MTS index.

FIG. 11 is an embodiment to which the present disclosure is applied, andillustrates a flowchart in which an inverse transform is performed basedon a transform-related parameter.

FIG. 12 is an embodiment to which the present disclosure is applied, andis a table illustrating that a transform set is assigned to eachintra-prediction mode in an NSST.

FIG. 13 is an embodiment to which the present disclosure is applied, andillustrates a calculation flow diagram for Givens rotation.

FIG. 14 is an embodiment to which the present disclosure is applied, andillustrates one round configuration in a 4×4 NSST composed of a Givensrotation layer and permutations.

FIG. 15 is an embodiment to which the present disclosure is applied, andis a block diagram for describing operations of a forward reducedtransform and an inverse reduced transform.

FIG. 16 is an embodiment to which the present disclosure is applied, andis a diagram illustrating a process of performing an inverse scan from a64-th to a 17-th according to an inverse scan order.

FIG. 17 is an embodiment to which the present disclosure is applied, andillustrates three forward scan orders for a transform coefficient block.

FIG. 18 is an embodiment to which the present disclosure is applied, andillustrates the locations of valid transform coefficients and a forwardscan order for each 4×4 block when a diagonal scan is applied and a 4×4RST is applied in a top-left 4×8 block.

FIG. 19 is an embodiment to which the present disclosure is applied, andillustrates a case where valid transform coefficients of two 4×4 blocksare merged into a single 4×4 block when a diagonal scan is applied and a4×4 RST is applied in a top-left 4×8 block.

FIG. 20 is an embodiment to which the present disclosure is applied, andillustrates a flowchart in which a video signal is encoded based on areduced secondary transform (RST).

FIG. 21 is an embodiment to which the present disclosure is applied, andillustrates a flowchart in which a video signal is decoded based on areduced secondary transform (RST).

FIG. 22 illustrates an example of a transform block and a top-left 4×4region according to an embodiment of the present disclosure.

FIG. 23 illustrates an example of a table for determining a transformtype in a horizontal direction and a vertical direction according to aprediction mode, wherein FIG. 23a illustrates a transform pair to whicha residual generated through an intra prediction is applied, and FIG.23b illustrates a transform pair to which a residual generated throughan inter prediction is applied.

FIG. 24 illustrates a flowchart of a method of encoding a video signalaccording to an embodiment of the present disclosure.

FIG. 25 illustrates an example of a transform process according to thesize of a transform block in an encoding process of a video signalaccording to an embodiment of the present disclosure.

FIG. 26 illustrates a flowchart of a process of decoding a video signalaccording to an embodiment of the present disclosure.

FIG. 27 illustrates an example of a transform process according to thesize of a transform block in a process of decoding a video signalaccording to an embodiment of the present disclosure.

FIG. 28 illustrates an example of a video coding system as an embodimentto which the present disclosure is applied.

FIG. 29 illustrates an example of a video streaming system as anembodiment to which the present disclosure is applied.

MODE FOR INVENTION

Reference will now be made in detail to embodiments of the presentdisclosure, examples of which are illustrated in the accompanyingdrawings. A detailed description to be disclosed below together with theaccompanying drawing is to describe exemplary embodiments of the presentdisclosure and not to describe a unique embodiment for carrying out thepresent disclosure. The detailed description below includes details toprovide a complete understanding of the present disclosure. However,those skilled in the art know that the present disclosure can be carriedout without the details.

In some cases, in order to prevent a concept of the present disclosurefrom being ambiguous, known structures and devices may be omitted orillustrated in a block diagram format based on core function of eachstructure and device.

Further, although general terms widely used currently are selected asthe terms in the present disclosure as much as possible, a term that isarbitrarily selected by the applicant is used in a specific case. Sincethe meaning of the term will be clearly described in the correspondingpart of the description in such a case, it is understood that thepresent disclosure will not be simply interpreted by the terms only usedin the description of the present disclosure, but the meaning of theterms should be figured out.

Specific terminologies used in the description below may be provided tohelp the understanding of the present disclosure. Furthermore, thespecific terminology may be modified into other forms within the scopeof the technical concept of the present disclosure. For example, asignal, data, a sample, a picture, a frame, a block, etc may be properlyreplaced and interpreted in each coding process.

In the present disclosure, a ‘processing unit’ refers to a unit on whichencoding/decoding process such as prediction, transform and/orquantization is performed. The processing unit may also be interpretedas the meaning including a unit for a luminance component and a unit fora chrominance component. For example, the processing unit may correspondto a block, a coding unit (CU), a prediction unit (PU) or a transformunit (TU).

The processing unit may also be interpreted as a unit for a luminancecomponent or a unit for a chrominance component. For example, theprocessing unit may correspond to a coding tree block (CTB), a codingblock (CB), a prediction unit (PU) or a transform block (TB) for theluminance component. Alternatively, the processing unit may correspondto a CTB, a CB, a PU or a TB for the chrominance component. Theprocessing unit is not limited thereto and may be interpreted as themeaning including a unit for the luminance component and a unit for thechrominance component.

In addition, the processing unit is not necessarily limited to a squareblock and may be configured in a polygonal shape having three or morevertexes.

In the present disclosure, a pixel is commonly called a sample. Inaddition, using a sample may mean using a pixel value or the like.

FIG. 1 is a schematic block diagram of an encoder in which encoding of avideo signal is performed as an embodiment to which the presentdisclosure is applied.

Referring to FIG. 1, the encoder 100 may be configured to include animage division unit 110, a transform unit 120, a quantization unit 130,a dequantization unit 140, an inverse transform unit 150, a filteringunit 160, a decoded picture buffer (DPB) 170, an inter-prediction unit180, an intra-prediction unit 185, and an entropy encoding unit 190 . .. .

The image division unit 110 may divide an input image (or picture orframe) input into the encoder 100 into one or more processing units. Forexample, the processing unit may be a Coding Tree Unit (CTU), a CodingUnit (CU), a Prediction Unit (PU), or a Transform Unit (TU). In thefollowing description, a transform unit (TU), that is, a unit on which atransform is performed is collectively referred to as a transform block.

However, the terms are only used for the convenience of description ofthe present disclosure and the present disclosure is not limited to thedefinition of the terms. In addition, in the present disclosure, for theconvenience of the description, the term coding unit is used as a unitused in encoding or decoding a video signal, but the present disclosureis not limited thereto and may be appropriately interpreted according tothe present disclosure.

The encoder 100 subtracts a prediction signal output from theinter-prediction unit 180 or the intra-prediction unit 185 from theinput image signal to generate a residual signal and the generatedresidual signal is transmitted to the transform unit 120.

The transform unit 120 may generate a transform coefficient by applyinga transform technique to the residual signal. A transform process may beapplied to a quadtree structure square block and a block (square orrectangle) divided by a binary tree structure, a ternary tree structure,or an asymmetric tree structure.

The transform unit 120 may perform a transform based on a plurality oftransforms (or transform combinations), and the transform scheme may bereferred to as multiple transform selection (MTS). The MTS may also bereferred to as an Adaptive Multiple Transform (AMT) or an EnhancedMultiple Transform (EMT).

The MTS (or AMT or EMT) may refer to a transform scheme performed basedon a transform (or transform combinations) adaptively selected from theplurality of transforms (or transform combinations).

A plurality of transforms (or transform combinations) may be determinedbased on a kernel of a discrete cosine transform (DCT) or discrete sinetransform (DST) type as illustrated in FIGS. 6a and 6b of the presentdisclosure. In the present disclosure, the transform type may beexpressed like DCT-Type 2, DCT-II, DCT-2, DCT2, for example, and isgenerally indicated as DCT-2 in the following description.

The transform unit 120 according to an embodiment of the presentdisclosure may perform the following embodiments.

The transform unit 120 may be configured to apply a forward primarytransform to each transform block including residual samples in each ofa horizontal direction and vertical direction, omit the application of aforward secondary transform to a transform block to which the primarytransform has been applied when the size of the transform block issmaller than or equal to 4×4, and apply a forward non-separablesecondary transform to the transform block when the size of a transformblock is greater than 4×4. Furthermore, the transform unit 120 may beconfigured to apply the forward non-separable secondary transform to atop-left 4×4 region of the transform block when the size of a transformblock is greater than 4×4.

The quantization unit 130 may quantize the transform coefficient andtransmits the quantized transform coefficient to the entropy encodingunit 190 and the entropy encoding unit 190 may entropy-code a quantizedsignal and output the entropy-coded quantized signal as a bitstream.

Although the transform unit 120 and the quantization unit 130 aredescribed as separate functional units, the present disclosure is notlimited thereto and may be combined into one functional unit. Thedequantization unit 140 and the inverse transform unit 150 may also besimilarly combined into one functional unit.

A quantized signal output from the quantization unit 130 may be used forgenerating the prediction signal. For example, inverse quantization andinverse transform are applied to the quantized signal through thedequantization unit 140 and the inverse transform unit 1850 in a loop toreconstruct the residual signal. The reconstructed residual signal isadded to the prediction signal output from the inter-prediction unit 180or the intra-prediction unit 185 to generate a reconstructed signal.

Meanwhile, deterioration in which a block boundary is shown may occurdue to a quantization error which occurs during such a compressionprocess. Such a phenomenon is referred to as blocking artifacts and thisis one of key elements for evaluating an image quality. A filteringprocess may be performed in order to reduce the deterioration. Blockingdeterioration is removed and an error for the current picture is reducedthrough the filtering process to enhance the image quality.

The filtering unit 160 applies filtering to the reconstructed signal andoutputs the applied reconstructed signal to a reproduction device ortransmits the output reconstructed signal to the decoded picture buffer170. The inter-prediction unit 170 may use the filtered signaltransmitted to the decoded picture buffer 180 as the reference picture.As such, the filtered picture is used as the reference picture in theinter prediction mode to enhance the image quality and the encodingefficiency.

The decoded picture buffer 170 may store the filtered picture in orderto use the filtered picture as the reference picture in theinter-prediction unit 180.

The inter-prediction unit 180 performs a temporal prediction and/orspatial prediction in order to remove temporal redundancy and/or spatialredundancy by referring to the reconstructed picture. In this case,since the reference picture used for prediction is a transformed signalthat is quantized and dequantized in units of the block at the time ofencoding/decoding in the previous time, blocking artifacts or ringingartifacts may exist.

Accordingly, the inter-prediction unit 180 may interpolate a signalbetween pixels in units of a sub-pixel by applying a low-pass filter inorder to solve performance degradation due to discontinuity orquantization of such a signal. In this case, the sub-pixel means avirtual pixel generated by applying an interpolation filter and aninteger pixel means an actual pixel which exists in the reconstructedpicture. As an interpolation method, linear interpolation, bi-linearinterpolation, wiener filter, and the like may be adopted.

An interpolation filter is applied to the reconstructed picture toenhance precision of prediction. For example, the inter-prediction unit180 applies the interpolation filter to the integer pixel to generate aninterpolated pixel and the prediction may be performed by using aninterpolated block constituted by the interpolated pixels as theprediction block.

Meanwhile, the intra-prediction unit 185 may predict the current blockby referring to samples in the vicinity of a block which is to besubjected to current encoding. The intra-prediction unit 185 may performthe following process in order to perform the intra prediction. First, areference sample may be prepared, which is required for generating theprediction signal. In addition, the prediction signal may be generatedby using the prepared reference sample. Thereafter, the prediction modeis encoded. In this case, the reference sample may be prepared throughreference sample padding and/or reference sample filtering. Since thereference sample is subjected to prediction and reconstructionprocesses, a quantization error may exist. Accordingly, a referencesample filtering process may be performed with respect to eachprediction mode used for the intra prediction in order to reduce such anerror.

The prediction signal generated through the inter-prediction unit 180 orthe intra-prediction unit 185 may be used for generating thereconstructed signal or used for generating the residual signal.

FIG. 2 is a schematic block diagram of a decoder in which decoding of avideo signal is performed as an embodiment to which the presentdisclosure is applied.

Referring to FIG. 2, the decoder 200 may be configured to include aparsing unit (not illustrated), an entropy decoding unit 210, adequantization unit 220, an inverse transform unit 230, a filtering unit240, a decoded picture buffer (DPB) unit 250, an inter-prediction unit260, and an intra-prediction unit 265.

In addition, a reconstructed video signal output through the decoder 200may be reproduced through a reproduction device.

The decoder 200 may receive the signal output from the encoder 100 ofFIG. 1 and the received signal may be entropy-decoded through theentropy decoding unit 210.

The dequantization unit 220 obtains the transform coefficient from anentropy-decoded signal by using quantization step size information.

The inverse transform unit 230 inversely transforms the transformcoefficient to obtain the residual signal.

The inverse transform unit 230 according to an embodiment of the presentdisclosure may be configured to omit the application of an inversenon-separable secondary transform to a transform block when the size ofthe transform block is smaller than or equal to 4×4, apply an inversenon-separable secondary transform to the transform block when the sizeof the transform block is greater than 4×4, and apply an inverse primarytransform to the transform block separately in the horizontal directionand the vertical direction. The inverse transform unit 230 according toan embodiment of the present disclosure may be configured to apply aninverse non-separable secondary transform to a top-left 4×4 region ofthe transform block when the size of the transform block is greater than4×4.

Although the dequantization unit 220 and the inverse transform unit 230are described as separate functional units, the present disclosure isnot limited thereto and may be combined into one functional unit.

The obtained residual signal is added to the prediction signal outputfrom the inter-prediction unit 260 or the intra-prediction unit 265 togenerate the reconstructed signal.

The filtering unit 240 applies filtering to the reconstructed signal andoutputs the applied reconstructed signal to a generation device ortransmits the output reconstructed signal to the decoded picture bufferunit 250. The inter-prediction unit 250 may use the filtered signaltransmitted to the decoded picture buffer unit 260 as the referencepicture.

In the present disclosure, the embodiments described in the transformunit 120 and the respective functional units of the encoder 100 may beequally applied to the inverse transform unit 230 and the correspondingfunctional units of the decoder, respectively.

FIG. 3 are embodiments to which the present disclosure may be applied,FIG. 3a is a diagram for describing a block split structure based on aquadtree (hereinafter referred to as a “QT”), FIG. 3b is a diagram fordescribing a block split structure based on a binary tree (hereinafterreferred to as a “BT”), FIG. 3c is a diagram for describing a blocksplit structure based on a ternary tree (hereinafter referred to as a“TT”), and FIG. 3d is a diagram for describing a block split structurebased on an asymmetric tree (hereinafter referred to as an “AT”).

In video coding, one block may be split based on a quadtree (QT).

Furthermore, one subblock split by the QT may be further splitrecursively using the QT. A leaf block that is no longer QT split may besplit using at least one method of a binary tree (BT), a ternary tree(TT) or an asymmetric tree (AT). The BT may have two types of splits ofa horizontal BT (2N×N, 2N×N) and a vertical BT (N×2N, N×2N). The TT mayhave two types of splits of a horizontal TT (2N×½N, 2N×N, 2N×½N) and avertical TT (½N×2N, N×2N, ½N×2N). The AT may have four types of splitsof a horizontal-up AT (2N×½N, 2N×3/2N), a horizontal-down AT (2N×3/2N,2N×½N), a vertical-left AT (½N×2N, 3/2N×2N), and a vertical-right AT(3/2N×2N, ½N×2N). Each BT, TT, or AT may be further split recursivelyusing the BT, TT, or AT.

FIG. 3a illustrates an example of a QT split. A block A may be splitinto four subblocks A0, A1, A2, and A3 by a QT. The subblock A1 may besplit into four subblocks B0, B1, B2, and B3 by a QT.

FIG. 3b illustrates an example of a BT split. A block B3 that is nolonger split by a QT may be split into vertical BTs C0 and C1 orhorizontal BTs D0 and D1. As in the block C0, each subblock may befurther split recursively like the form of horizontal BTs E0 and E1 orvertical BTs F0 and F1.

FIG. 3c illustrates an example of a TT split. A block B3 that is nolonger split by a QT may be split into vertical TTs C0, C1, and C2 orhorizontal TTs D0, D1, and D2. As in the block C1, each subblock may befurther split recursively like the form of horizontal TTs E0, E1, and E2or vertical TTs F0, F1, and F2.

FIG. 3d illustrates an example of an AT split. A block B3 that is nolonger split by a QT may be split into vertical ATs C0 and C1 orhorizontal ATs D0 and D1. As in the block C1, each subblock may befurther split recursively like the form of horizontal ATs E0 and E1 orvertical TTs F0 and F1.

Meanwhile, BT, TT, and AT splits may be split together. For example, asubblock split by a BT may be split by a TT or AT. Furthermore, asubblock split by a TT may be split by a BT or AT. A subblock split byan AT may be split by a BT or TT. For example, after a horizontal BTsplit, each subblock may be split into vertical BTs or after a verticalBT split, each subblock may be split into horizontal BTs. The two typesof split methods are different in a split sequence, but have the samefinally split shape.

Furthermore, if a block is split, the sequence that the block issearched may be defined in various ways. In general, the search isperformed from left to right or from top to bottom. To search a blockmay mean a sequence for determining whether to split an additional blockof each split subblock or may mean a coding sequence of each subblock ifa block is no longer split or may mean a search sequence wheninformation of another neighbor block is referred in a subblock.

A transform may be performed for each processing unit (or transformunit) divided by a division structure, such as FIGS. 3a to 3d . Inparticular, a division may be performed for each row direction and eachcolumn direction, and a transform matrix may be applied. According to anembodiment of the present disclosure, a different transform type may beused based on the length of a processing unit (or transform unit) in therow direction or column direction.

FIGS. 4 and 5 are embodiments to which the present disclosure isapplied. FIG. 4 illustrates a schematic block diagram of a transform andquantization unit 120/130 and a dequantization and transform unit140/150 within the encoder, and FIG. 5 illustrates a schematic blockdiagram of a dequantization and transform unit 220/230 within thedecoder.

Referring to FIG. 4, the transform and quantization unit 120/130 mayinclude a primary transform unit 121, a secondary transform unit 122 andthe quantization unit 130. The dequantization and transform unit 140/150may include the dequantization unit 140, an inverse secondary transformunit 151 and an inverse primary transform unit 152.

Referring to FIG. 5, the dequantization and transform unit 220/230 mayinclude the dequantization unit 220, an inverse secondary transform unit231 and an inverse primary transform unit 232.

In the present disclosure, when a transform is performed, the transformmay be performed through a plurality of steps. For example, as in FIG.4, two steps of a primary transform and a secondary transform may beapplied or more transform steps may be used according to an algorithm.In this case, the primary transform may be referred to as a coretransform.

The primary transform unit 121 may apply a primary transform on aresidual signal. In this case, the primary transform may be pre-definedin a table form in the encoder and/or the decoder.

Furthermore, combinations of several transform types (DCT-2, DST-7,DCT-8) of MTS may be used for the primary transform. For example, atransform type may be determined as in the tables illustrated in FIGS.6a and 6 b.

The secondary transform unit 122 may apply a secondary transform to aprimary-transformed signal. In this case, the secondary transform may bepre-defined as a table in the encoder and/or the decoder.

According to an embodiment of the present disclosure, the secondarytransform unit 122 may be configured to omit the application of aforward non-separable secondary transform (NSST) to a transform block towhich a primary transform has been applied when the size of thetransform block is smaller than or equal to 4×4 and to apply the forwardnon-separable secondary transform to the transform block when the sizeof the transform block is greater than 4×4. Furthermore, the secondarytransform unit 122 may be configured to apply the forward non-separablesecondary transform (NSST) to a top-left 4×4 region of the transformblock.

In one embodiment, the non-separable secondary transform (NSST) may beconditionally applied to the secondary transform. For example, the NSSTmay be applied to only an intra prediction block, and may have atransform set which may be applied to each prediction mode group.

In this case, the prediction mode group may be configured based onsymmetry with respect to a prediction direction. For example, sinceprediction mode 52 and prediction mode 16 are symmetrical based onprediction mode 34 (diagonal direction), the same transform set may beapplied by forming one group. In this case, when the transform forprediction mode 52 is applied, input data is transposed and then appliedbecause prediction mode 52 has the same transform set as prediction mode16.

Meanwhile, since the symmetry for the direction does not exist in thecase of a planar mode and a DC mode, each mode has a different transformset and the corresponding transform set may include two transforms. Inrespect to the remaining direction modes, each transform set may includethree transforms.

The quantization unit 130 may perform quantization on asecondary-transformed signal.

The inverse quantization unit and the inverse transform unit 140/150inversely perform the aforementioned process, and a redundantdescription thereof is omitted.

FIG. 5 is a schematic block diagram of a dequantization unit 220 and aninverse transform unit 230 in a decoder.

Referring to FIG. 5 above, the dequantization and inverse transformunits 220 and 230 may include a dequantization unit 220, an inversesecondary transform unit 231, and an inverse primary transform unit 232.

The dequantization unit 220 obtains the transform coefficient from anentropy-decoded signal by using quantization step size information.

The inverse secondary transform unit 231 performs an inverse secondarytransform for the transform coefficients. In this case, the inversesecondary transform represents an inverse transform of the secondarytransform described in FIG. 4.

The inverse secondary transform unit 231 according to an embodiment ofthe present disclosure may be configured to omit the application of aninverse non-separable secondary transform to a transform block when thesize of the transform block is smaller than or equal to 4×4, apply theinverse non-separable secondary transform to the transform block whenthe size of a transform block is greater than 4×4, and apply an inverseprimary transform to the transform block separately in the horizontaldirection and the vertical direction. The inverse secondary transformunit 231 according to an embodiment of the present disclosure may beconfigured to apply the inverse non-separable secondary transform to atop-left 4×4 region of the transform block when the size of thetransform block is greater than 4×4.

The inverse primary transform unit 232 performs an inverse primarytransform for the inverse secondary transformed signal (or block) andobtains the residual signal. In this case, the inverse primary transformrepresents the inverse transform of the primary transform described inFIG. 4.

In one embodiment, combinations of several transforms (DCT-2, DST-7,DCT-8) of MTS may be applied to the primary transform. For example, atransform type may be determined as in the tables illustrated in FIGS.6a and 6b O|.

FIGS. 6a and 6b illustrate examples of tables for determining atransform type for a horizontal direction and a vertical direction foreach prediction mode. FIG. 6a illustrates an example of the table fordetermining transform types in the horizontal/vertical direction in anintra-prediction mode. FIG. 6b illustrates an example of the table fordetermining transform types for the horizontal/vertical direction in aninter-prediction mode. FIGS. 6a and 6b are examples of combinationtables for determining transform types, illustrates MTS combinationsapplied to a joint exploration model (JEM). Another combination may alsobe used. For example, the table of FIG. 6b may be used for both anintra-prediction and an inter-prediction. Examples applied to the JEMare basically described with reference to FIGS. 6a and 6 b.

In the JEM, the application of MTS may become on/off in a block unit (inthe case of HEVC, in a CU unit) because a syntax element calledEMT_CU_flag (or MTS_CU_flag) is introduced. That is, in theintra-prediction mode, when MTS_CU_flag is 0, DCT-2 or DST-7 (for a 4×4block) in the existing high efficiency video coding (HEVC) is used. WhenMTS_CU_flag is 1, an MTS combination proposed in FIG. 6a is used. Apossible MTS combination may be different depending on anintra-prediction mode as in FIG. 6a . For example, a total of possiblefour combinations are permitted because DST-7 and DCT-5 are used in thehorizontal direction and DST-7 and DCT-8 are used in the verticaldirection, with respect to 14, 15, 16, 17, 18, 19, 20, 21, 22 modes.Accordingly, there is a need for signaling of information on which oneof the four combinations is used. One of the four combinations isselected through MTS_TU_index having two bits. FIG. 6b illustrates MTScombinations which may be applied in an inter-prediction mode. Unlike inFIG. 6a , a combination which is possible based on only DST-7 and DCT-8is determined. According to an embodiment of the present disclosure,EMT_CU_flag may be used instead of MTS_CU_flag.

FIG. 7 is an embodiment to which the present disclosure is applied, andis a flowchart illustrating an encoding process in which MTS isperformed.

In the present disclosure, basically, embodiments in which transformsare applied in the horizontal direction and the vertical direction arebasically described. However, a transform combination may be configuredwith a non-separable transform.

Alternatively, a mixture of separable transforms and non-separabletransforms may be configured. In this case, if the non-separabletransform is used, transform selection for each row/column or selectionfor each horizontal/vertical direction becomes unnecessary. Thetransform combinations of FIG. 6a or 6 b may be used only when theseparable transform is selected.

Furthermore, methods proposed in the present disclosure may be appliedregardless of a primary transform or a secondary transform. That is,there is no restriction in that the methods should be applied to any oneof the primary transform and the secondary transform, and both theprimary transform and the secondary transform may be applied. In thiscase, the primary transform may mean a transform for first transforminga residual block. The secondary transform may mean a transform forapplying a transform to a block generated as a result of the primarytransform. According to an embodiment of the present disclosure, whenthe size of a transform block is smaller than or equal to 4×4, only aprimary transform may be applied in the state in which a secondarytransform has been omitted. When the size of the transform block isgreater than 4×4, the secondary transform may be applied to a top-left4×4 region.

First, the encoder 100 may determine a transform configuration groupcorresponding to a current block (S710). In this case, the transformconfiguration group may be composed of combinations, such as FIGS. 6aand 6 b.

The encoder may perform a transform on available candidate transformcombinations within the transform configuration group (S720).

As a result of the execution of the transform, the encoder may determineor select a transform combination having the smallest rate distortion(RD) cost (S730).

The encoder may encode a transform combination index corresponding tothe selected transform combination (S740).

FIG. 8 is an embodiment to which the present disclosure is applied, andis a flowchart illustrating a decoding process in which MTS isperformed.

First, the decoder 200 may determine a transform configuration group fora current block (S810). The decoder 200 may parse (or obtain) atransform combination index from a video signal. In this case, thetransform combination index may correspond to one of a plurality oftransform combinations within the transform configuration group (S820).For example, the transform configuration group may include DST-4, DCT-4,DST-7 and DCT-8. The transform combination index may be called an MTSindex. In an embodiment, the transform configuration group may beconfigured based on at least one of a prediction mode, a block size or ablock shape of the current block.

The decoder 100 may derive a transform combination corresponding to atransform combination index (S830). In this case, the transformcombination may be composed of a horizontal transform and a verticaltransform, and may include at least one of DCT-2, DST-7 or DCT-8.Furthermore, the transform combinations described with reference to FIG.6a or 6 b may be used for the transform combination. That is, in thepresent disclosure, a configuration based on another transformcombination according to another embodiment is possible.

The decoder 100 may perform an inverse transform on the current blockbased on the derived transform combination (S840). If the transformcombination is configured with a row (horizontal) transform and a column(vertical) transform, after the row (horizontal) transform is firstapplied, the column (vertical) transform may be applied, but the presentdisclosure is not limited thereto. If the transform combination isconfigured in an opposite way or configured with non-separabletransforms, a non-separable transform may be applied.

In an embodiment, if a vertical transform or a horizontal transform isDST-7 or DCT-8, an inverse transform of DST-7 or an inverse transform ofDST-8 may be applied for each column, and then applied for each row.Furthermore, in the vertical transform or the horizontal transform, adifferent transform may be applied for each row and/or for each column.

In an embodiment, a transform combination index may be obtained based onan MTS flag indicating whether MTS is performed. That is, the transformcombination index may be obtained only when MTS is performed based on anMTS flag. Furthermore, the decoder 100 may check whether the number ofnon-zero coefficients is greater than a threshold value. In this case,the transform combination index may be obtained only when the number ofnon-zero coefficients is greater than the threshold value.

In an embodiment, the MTS flag or the MTS index may be defined in atleast one level of a sequence, a picture, a slice, a block, a codingunit, a transform unit, or a prediction unit.

In an embodiment, an inverse transform may be applied only when both thewidth and height of a transform unit are 32 or less.

Meanwhile, in another embodiment, the process of determining thetransform configuration group and the process of parsing the transformcombination index may be simultaneously performed. Alternatively, stepS810 may be pre-configured in the encoder 100 and/or the decoder 200 andomitted.

FIG. 9 is an embodiment to which the present disclosure is applied, andis a flowchart for describing a process of encoding an MTS flag and anMTS index.

The encoder 100 may determine whether MTS is applied to a current block(S910).

If the MTS is applied, the encoder 100 may encode the MTS flag=1 (S920).

Furthermore, the encoder 100 may determine an MTS index based on atleast one of a prediction mode, horizontal transform, or verticaltransform of the current block (S930). In this case, the MTS index meansan index indicative of any one of a plurality of transform combinationsfor each intra-prediction mode, and the MTS index may be transmitted foreach transform unit.

When the MTS index is determined, the encoder 100 may encode the MTSindex determined at step S930 (S940).

Meanwhile, if the MTS is not applied, the encoder 100 may encode MTSflag=0 (S950).

FIG. 10 is an embodiment to which the present disclosure is applied, andis a flowchart for illustrating a decoding process of applying ahorizontal transform or a vertical transform to a row or column based onan MTS flag and an MTS index.

The decoder 200 may parse an MTS flag from a bit stream (S1010). In thiscase, the MTS flag may indicate whether MTS is applied to a currentblock.

The decoder 200 may check whether the MTS is applied to the currentblock based on the MTS flag (S1020). For example, the decoder 200 maycheck whether the MTS flag is 1.

When the MTS flag is 1, the decoder 200 may check whether the number ofnon-zero coefficients is greater than (or equal to or greater than) athreshold value (S1030). For example, the threshold value for the numberof transform coefficients may be set to 2. The threshold value may beset based on a block size or the size of a transform unit

When the number of non-zero coefficients is greater than the thresholdvalue, the decoder 200 may parse an MTS index (S1040). In this case, theMTS index means an index indicative of any one of a plurality oftransform combinations for each intra-prediction mode or eachinter-prediction mode. The MTS index may be transmitted for eachtransform unit. Furthermore, the MTS index may mean an index indicativeof any one transform combination defined in a pre-configured transformcombination table. In this case, the pre-configured transformcombination table may mean the tables of FIG. 6a or 6 b, but the presentdisclosure is not limited thereto.

The decoder 100 may derive or determine a horizontal transform and avertical transform based on at least one of the MTS index or aprediction mode (S1050). Furthermore, the decoder 100 may derive atransform combination corresponding to the MTS index. For example, thedecoder 100 may derive or determine a horizontal transform and avertical transform corresponding to the MTS index.

Meanwhile, when the number of non-zero coefficients is not greater thanthe threshold value, the decoder 200 may apply a pre-configured verticalinverse transform for each column (S1060). For example, the verticalinverse transform may be an inverse transform of DST-7. Furthermore, thevertical inverse transform may be an inverse transform of DST-8

Furthermore, the decoder may apply a pre-configured horizontal inversetransform for each row (S1070). For example, the horizontal inversetransform may be an inverse transform of DST-7. Furthermore, thehorizontal inverse transform may be an inverse transform of DST-8.

That is, when the number of non-zero coefficients is not greater thanthe threshold value, a transform type pre-configured in the encoder 100or the decoder 200 may be used. For example, not a transform typedefined in a transform combination table, such as FIG. 6a or 6 b, but atransform type (e.g., DCT-2) which is commonly used may be used.

Meanwhile, when the MTS flag is 0, the decoder 200 may apply apre-configured vertical inverse transform for each column (S1080). Forexample, the vertical inverse transform may be an inverse transform ofDCT-2.

Furthermore, the decoder 200 may apply a pre-configured horizontalinverse transform for each row (S1090). For example, the horizontalinverse transform may be an inverse transform of DCT-2. That is, whenthe MTS flag is 0, a transform type pre-configured in the encoder or thedecoder may be used. For example, not a transform type defined in atransform combination table, such as FIG. 6a or 6 b, but a transformtype which is commonly used may be used.

FIG. 11 is an embodiment to which the present disclosure is applied, andillustrates a flowchart in which an inverse transform is performed basedon a transform-related parameter.

The decoder 200 to which the present disclosure is applied may obtainsps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1110). Inthis case, sps_mts_intra_enabled_flag indicates whether tu_mts_flag ispresent in a residual coding syntax a coding unit to which anintra-prediction is applied (intra-coding unit). For example, whensps_mts_intra_enabled_flag=0, tu_mts_flag is not present in the residualcoding syntax of the intra-coding unit. Whensps_mts_intra_enabled_flag=0, tu_mts_flag is present in the residualcoding syntax of the intra-coding unit. Furthermore,sps_mts_inter_enabled_flag indicates whether tu_mts_flag is present in aresidual coding syntax of a coding unit to which an inter-prediction isapplied (inter-coding unit). For example, whensps_mts_inter_enabled_flag=0, tu_mts_flag is not present in the residualcoding syntax of the inter-coding unit. Whensps_mts_inter_enabled_flag=0, tu_mts_flag is present in the residualcoding syntax of the inter-coding unit.

The decoder 200 may obtain tu_mts_flag based onsps_mts_intra_enabled_flag or sps_mts_inter_enabled_flag (S1120). Forexample, when sps_mts_intra_enabled_flag=1 orsps_mts_inter_enabled_flag=1, the decoder 200 may obtain tu_mts_flag. Inthis case, tu_mts_flag indicates whether MTS is applied to a residualsample of a luminance transform unit. For example, when tu_mts_flag=0,the MTS is not applied to a residual sample of the luminance transformunit. When tu_mts_flag=1, the MTS is applied to a residual sample of theluminance transform unit. At least one of embodiments described in thepresent disclosure may be applied with respect to Tu_mts_flag=1.

The decoder 200 may obtain mts_idx based on tu_mts_flag (S1130). Forexample, when tu_mts_flag=1, the decoder may obtain mts_idx. In thiscase, mts_idx indicates whether which transform kernel is applied toluminance residual samples according to the horizontal and/or verticaldirection of a current transform block. For example, at least one of theembodiments of the present disclosure may be applied to mts_idx. As adetailed example, at least one of the embodiments of FIGS. 6a and 6b,44a , or 44 b may be applied.

The decoder 200 may derive a transform kernel corresponding to mts_idx(S1140). For example, the transform kernel corresponding to mts_idx maybe divided and defined into a horizontal transform and a verticaltransform.

For another example, different transform kernels may be applied to thehorizontal transform and the vertical transform, but the presentdisclosure is not limited thereto. The same transform kernel may beapplied to the horizontal transform and the vertical transform.

In an embodiment, mts_idx may be defined like Table 1.

TABLE 1 mts_idx[x0][y0] trTypeHor trTypeVer 0 0 0 1 1 1 2 2 1 3 1 2 4 22

Furthermore, the decoder 200 may perform an inverse transform based onthe transform kernel derived at step S1140 (S1150).

In FIG. 11, an embodiment in which in order to determine whether toapply MTS, tu_mts_flag is obtained, mts_idx is obtained based on theobtained tu_mts_flag value, and a transform kernel is determined hasbeen basically described, but the present disclosure is not limitedthereto. For example, the decoder 200 may determine a transform kernelby directly parsing mts_idx without parsing tu_mts_flag. In this case,Table 1 may be used. That is, when the mts_idx value indicates 0, DCT-2may be applied in the horizontal/vertical direction. When the mts_idxvalue indicates a value other than 0, DST-7 or DCT-8 may be appliedbased on the mts_idx value.

In another embodiment of the present disclosure, a decoding process ofperforming a transform process is described.

The decoder 200 may check a transform size (nTbS). In this case, thetransform size (nTbS) may be a variable indicative of a horizontalsample size of scaled transform coefficients.

The decoder 200 may check a transform kernel type (trType). In thiscase, the transform kernel type (trType) may be a variable indicative ofthe type of transform kernel, and various embodiments of the presentdisclosure may be applied. The transform kernel type (trType) mayinclude a horizontal transform kernel type (trTypeHor) and a verticaltransform kernel type (trTypeVer).

Referring to Table 1, the transform kernel type (trType) may indicateDCT-2 when it is 0, may indicate DST-7 when it is 1, and may indicateDCT-8 when it is 2.

The decoder 200 may perform a transform matrix multiplication based onat least one of a transform size (nTbS) or a transform kernel type.

For another example, when the transform kernel type is 1 and thetransform size is 4, a previously determined transform matrix 1 may beapplied when a transform matrix multiplication is performed.

For another example, when the transform kernel type is 1 and thetransform size is 8, a previously determined transform matrix 2 may beapplied when a transform matrix multiplication is performed.

For another example, when the transform kernel type is 1 and thetransform size is 16, a previously determined transform matrix 3 may beapplied when a transform matrix multiplication is performed.

For another example, when the transform kernel type is 1 and thetransform size is 32, a previously defined transform matrix 4 may beapplied.

Likewise, when the transform kernel type is 2 and the transform size is4, 8, 16, and 32, a previously defined transform matrices 5, 6, 7, and 8may be applied, respectively.

In this case, each of the previously defined transform matrices 1 to 8may correspond to any one of various types of transform matrices. Forexample, the transform matrices of types illustrated in FIG. 6 may beapplied.

The decoder 200 may derive a transform sample based on a transformmatrix multiplication.

The embodiments may be used, but the present disclosure is not limitedthereto. The above embodiment and other embodiments of the presentdisclosure may be combined and used.

FIG. 12 is an embodiment to which the present disclosure is applied, andis a table illustrating that a transform set is assigned to eachintra-prediction mode in an NSST.

The secondary transform unit 122 may apply a secondary transform to aprimary transformed signal. In this case, the secondary transform may bepre-defined as a table in the encoder 100 and/or the decoder 200.

In an embodiment, an NSST may be conditionally applied to the secondarytransform. For example, the NSST is applied only in the case of anintra-prediction block, and may have an applicable transform set foreach prediction mode group.

According to an embodiment of the present disclosure, the NSST may beomitted when the size of a transform block corresponds to 4×4, and theNSST may be applied to only a top-left 4×4 region when the size of thetransform block is greater than 4×4.

In this case, the prediction mode group may be configured based onsymmetry for a prediction direction. For example, a prediction mode 52and a prediction mode 16 are symmetrical to each other with respect to aprediction mode 34 (diagonal direction), and thus may form one group, sothe same transform set may be applied to the prediction mode 52 and theprediction mode 16. In this case, when a transform for the predictionmode 52 is applied, input data is transposed and applied. The reason forthis is that the prediction mode 52 and the prediction mode 16 have thesame transform set.

Meanwhile, the planar mode and the DC mode have respective transformsets because symmetry for a direction is not present. A correspondingtransform set may be configured with two transforms. With respect to theremaining direction modes, three transforms may be configured for eachtransform set, but the present disclosure is not limited thereto. Eachtransform set may be configured with a plurality of transforms.

FIG. 13 is an embodiment to which the present disclosure is applied, andillustrates a calculation flow diagram for Givens rotation.

In another embodiment, the NSST is not applied to all of primarytransformed blocks, but may be applied to only a top-left 8×8 region.For example, if the size of a block is 8×8 or more, an 8×8 NSST isapplied. If the size of a block is less than 8×8, a 4×4 NSST is applied.In this case, the block is divided into 4×4 blocks and the 4×4 NSST isapplied to each of the blocks. According to an embodiment of the presentdisclosure, the NSST may be omitted when the size of a transform blockcorresponds to 4×4, and the NSST may be applied to only a top-left 4×4region when the size of the transform block is greater than 4×4.

As another embodiment, even in the case of 4×N/N×4 (N>=16), the 4×4 NSSTmay be applied.

Since both the 8×8 NSST and the 4×4 NSST follow a transformationcombination configuration described in the present disclosure and arethe non-separable transforms, the 8×8 NSST receives 64 data and outputs64 data and the 4×4 NSST has 16 inputs and 16 outputs.

Both the 8×8 NSST and the 4×4 NSST are configured by a hierarchicalcombination of Givens rotations. A matrix corresponding to one Givensrotation is shown in Equation 1 below and a matrix product is shown inEquation 2 below.

$\begin{matrix}{R_{\theta} = \begin{bmatrix}{\cos\;\theta} & {{- \sin}\;\theta} \\{\sin\;\theta} & {\cos\;\theta}\end{bmatrix}} & \left\lbrack {{Equation}\mspace{14mu} 1} \right\rbrack\end{matrix}$t _(m) =x _(m) cos θ−x _(n) sin θ

t _(n) =x _(m) sin θ−x _(n) cos θ  [Equation 2]

As illustrated in FIG. 13 above, since one Givens rotation rotates twodata, in order to process 64 data (for the 8×8 NSST) or 16 data (for the4×4 NSST), a total of 32 or 8 Givens rotations are required.

Therefore, a bundle of 32 or 8 is used to form a Givens rotation layer.Output data for one Givens rotation layer is transferred as input datafor a next Givens rotation layer through a determined permutation.

FIG. 14 illustrates one round configuration in 4×4 NSST constituted by agivens rotation layer and permutations as an embodiment to which thepresent disclosure is applied.

Referring to FIG. 14 above, it is illustrated that four Givens rotationlayers are sequentially processed in the case of the 4×4 NSST. Asillustrated in FIG. 14 above, the output data for one Givens rotationlayer is transferred as the input data for the next Givens rotationlayer through a determined permutation (i.e., shuffling).

As illustrated in FIG. 14 above, patterns to be permutated are regularlydetermined and in the case of the 4×4 NSST, four Givens rotation layersand the corresponding permutations are combined to form one round.

In the case of the 8×8 NSST, six Givens rotation layers and thecorresponding permutations form one round. The 4×4 NSST goes through tworounds and the 8×8 NSST goes through four rounds. Different rounds usethe same permutation pattern, but applied Givens rotation angles aredifferent. Accordingly, angle data for all Givens rotations constitutingeach transform need to be stored.

As a last step, one permutation is further finally performed on the dataoutput through the Givens rotation layers, and corresponding permutationinformation is stored separately for each transform. In forward NSST,the corresponding permutation is performed last and in inverse NSST, acorresponding inverse permutation is applied first on the contrarythereto.

In the case of the inverse NSST, the Givens rotation layers and thepermutations applied to the forward NSST are performed in the reverseorder and rotation is performed by taking a negative value even for anangle of each Givens rotation.

FIG. 15 is an embodiment to which the present disclosure is applied, andis a block diagram for describing operations of a forward reducedtransform and an inverse reduced transform.

Reduced Secondary Transform (RST)

Assuming that an orthogonal matrix indicative of one transform has anN×N form, a reduced transform (hereinafter referred to as an “RT”)leaves only R of N transform basis vectors (R<N). A matrix for a forwardRT for generating a transform coefficient is given as in Equation 3below.

$\begin{matrix}{T_{RxN} = \begin{bmatrix}t_{11} & t_{12} & t_{13} & \ldots & t_{1\; N} \\t_{21} & t_{22} & t_{23} & \; & t_{2\; N} \\\; & \vdots & \; & \ddots & \vdots \\t_{R\; 1} & t_{R\; 2} & t_{R\; 3} & \cdots & t_{R\; N}\end{bmatrix}} & \left\lbrack {{Equation}\mspace{14mu} 3} \right\rbrack\end{matrix}$

A matrix for an inverse RT is a transpose matrix of the forward RTmatrix. The application of the forward RT and the inverse RT isdiagrammed like FIG. 15.

Assuming that an RT is applied to a top-left 8×8 block of a transformblock that has experienced a primary transform, the RT may be named an8×8 reduced secondary transform (8×8 RST).

Assuming that an R value in Equation 3 is 16, a forward 8×8 RST has a16×64 matrix form, and an inverse 8×8 RST has a 64×16 matrix form.

Furthermore, the same transform set configuration as that in FIG. 12 maybe applied to an 8×8 RST. That is, a corresponding 8×8 RST may beapplied based on the transform set in FIG. 12.

As an embodiment, in FIG. 12, assuming that one transform set iscomposed of two or three transforms according to an intra predictionmode, one of a maximum of four transforms including a case where asecondary transform is not applied may be configured to be selected. Inthis case, the one transform may be considered as an identity matrix.

If indices 0, 1, 2, and 3 are assigned to four transforms, respectively,a corresponding transform may be designated by signaling a syntaxelement called an NSST index for each transform block. That is, in thecase of an NSST, an 8×8 NSST may be designated with respect to an 8×8top-left block based on an NSST index. In an RST configuration, the 8×8RST may be designated. Furthermore, in this case, a No. 0 index may beassigned as an identity matrix, that is, a case where a secondarytransform is not applied.

If a forward 8×8 RST, such as Equation 3, is applied, 16 valid transformcoefficients are generated. Accordingly, it may be considered that 64input data forming an 8×8 region is reduced to 16 output data. From atwo-dimensional region viewpoint, only a region of ¼ is filled withvalid transform coefficients. Accordingly, a 4×4 left-top region of FIG.16 may be filled with 16 output data obtained by applying a forward 8×8RST.

Meanwhile, the aforementioned RST may also be designated as a lowfrequency non-separable transform (LFNST).

FIG. 16 is an embodiment to which the present disclosure is applied, andis a diagram illustrating a process of performing an inverse scan from a64-th to a 17-th according to an inverse scan order.

FIG. 16 illustrates that scanning is performed on a 17-th coefficient toa 64-th coefficient (in a forward scan order) assuming that the forwardscan order starts from 1. However, FIG. 16 illustrates an inverse scanand illustrates that inverse scanning is performed on the 64-th to the17-th.

Referring to FIG. 16, a top-left 4×4 region is a region of interest(ROI) to which a valid transform coefficient is assigned, and theremaining regions are empted. That is, a value 0 may be assigned to theremaining regions by default.

If a valid transform coefficient not 0 is present in the ROI region ofFIG. 16, this means that an 8×8 RST is not applied. In this case,corresponding NSST index coding may be omitted.

In contrast, if a non-zero transform coefficient is not present inregions other than the ROI region of FIG. 16 (if the 8×8 RST is applied,0 is assigned to the regions other than the ROI), an NSST index may becoded because there is a possibility that the 8×8 RST has been applied.

As described above, conditional NSST index coding may be performed aftera residual coding process because it is necessary to check whether anon-zero transform coefficient is present.

According to an embodiment of the present disclosure, when the size of atransform block corresponds to 4×4, an NSST may be omitted. When thesize of the transform block is greater than 4×4, the NSST may be appliedto only a top-left 4×4 region.

FIG. 17 is an embodiment to which the present disclosure is applied, andillustrates three forward scan orders for a transform coefficient block.

Embodiment 1-1: RST to which 4×4 Block May be Applied

A non-separable transform to which one 4×4 block may be applied is a16×16 transform. That is, if data elements forming a 4×4 block are linedup in a row-first or column-first order, a 16×1 vector may be formed,and a non-separable transform may be applied to the corresponding 16×1vector.

A forward 16×16 transform is composed of 16 row direction transformbasis vectors. If an inner product is applied to the 16×1 vector andeach transform basis vector, a transform coefficient for the transformbasis vector is obtained. A process of obtaining all correspondingtransform coefficients for the 16 transform basis vectors is the same asthat a 16×16 non-separable transform matrix and an input 16×1 vector aremultiplied.

The transform coefficients obtained by the matrix multiplication have a16×1 vector form, and may have a different statistical characteristicfor each transform coefficient. For example, assuming that a 16×1transform coefficient vector is composed of a No. 0 element to a No. 15element, a dispersion of the No. 0 element may be greater than that ofthe No. 15 element. That is, an element that is located ahead may have agreat energy value because it has a great disperse value.

If an inverse 16×16 non-separable transform is applied to a 16×1transform coefficient, an original 4×4 block signal may bereconstructed. If a forward 16×16 non-separable transform is anorthonormal transform, a corresponding inverse 16×16 transform may beobtained through a transpose matrix for a forward 16×16 transform.

If an inverse 16×16 non-separable transform matrix is multiplied by a16×1 transform coefficient vector, data having a 16×1 vector form isobtained and arranged in a row-first or column-first order that has beenfirst applied, thereby being capable of obtaining a 4×4 block signal.

As described above, elements forming the 16×1 transform coefficientvector may have different statistical characteristics.

If transform coefficients located ahead (close to the No. 0 element)have greater energy, a signal much closer to an original signal can bereconstructed even if an inverse transform is applied to some transformcoefficients that first emerge without using all the transformcoefficients. For example, assuming that an inverse 16×16 non-separabletransform is composed of 16 column basis vectors, a 16×L matrix may beconfigured by leaving only L column basis vectors. Furthermore, afterimportant L transform coefficients are left (L×1 vector) among thetransform coefficients, if the 16×L matrix and the L×1 vector aremultiplied, a 16×1 vector not have a great error with the original input16×1 vector data may be reconstructed.

As a result, since only the L coefficients are used for datareconstruction, an L×1 transform coefficient vector not a 16×1 transformcoefficient vector has only to be calculated when a transformcoefficient is obtained. That is, after an L×16 transform is configuredby picking up L corresponding row direction transform vectors from aforward 16×16 non-separable transform matrix and is multiplied by a 16×1input vector, the important L transform coefficients may be obtained.

The L value has a range of 1<=L<16. In general, L vectors of 16transform basis vectors may be selected using a given method. However,from a coding and decoding viewpoint, to select transform basis vectorshaving great importance in terms of energy of a signal may beadvantageous from a coding efficiency viewpoint.

Embodiment 1-2: Configuration of Application Region of 4×4 RST andDeployment of Transform Coefficient

A 4×4 RST may be applied as a secondary transform. In this case, the 4×4RST may be two-dimensionally applied to a block a primary transform,such as DOT-2, has been applied. Assuming that the size of a block towhich a primary transform has been applied is N×N, in general, the sizeof the block to which the primary transform has been applied is greaterthan 4×4. Accordingly, when the 4×4 RST is applied to an N×N block, thefollowing two methods may be used.

As the first method, the 4×4 RST is not applied to the entire N×Nregion, but may be applied to some region. For example, the 4×4 RST maybe applied to a top-left M×M region only (M<=N).

As the second method, after a region to which a secondary transform willbe applied is divided into 4×4 blocks, the 4×4 RST may be applied toeach of the divided blocks.

As an embodiment, the two methods may be mixed. For example, after onlya top-left M×M region is divided into 4×4 blocks, a 4×4 RST may beapplied to the divided M×M regions.

As an embodiment, a secondary transform may be applied to only atop-left 8×8 region. When an N×N block is equal to or greater than 8×8,an 8×8 RST may be applied to the top-left 8×8 region. When the N×N blockis smaller than 8×8 (4×4, 8×4, 4×8), a top-left M×M region may bedivided into 4×4 blocks as in the second embodiment, and a 4×4 RST maybe applied to each of the 4×4 blocks. Furthermore, in the case of4×N/N×4 (N>=16), the 4×4 RST may be applied.

After the 4×4 RST is applied, when L (1<=L<16) transform coefficientsare generated, a degree of freedom for how the L transform coefficientswill be deployed occurs. However, when the transform coefficients areprocessed in a residual coding step, a predetermined order will bepresent. Accordingly, coding performance may be different depending onhow the L transform coefficients will be deployed in a two-dimensionalblock.

For example, in the case of HEVC residual coding, coding starts from alocation farthest from a DC location. This is for increasing codingperformance using the fact that a coefficient value that has experiencedquantization is 0 or close to 0 as it becomes farther from the DClocation.

Accordingly, it may be advantageous to deploy more importantcoefficients having higher energy than L transform coefficients so thatthey are coded later in order of residual coding in terms of codingperformance.

FIG. 17 illustrates three forward scan orders of a 4×4 transform block(coefficient group (CG)) unit which is applied in HEVC. Residual codingfollows an inverse order of the scan order in FIG. 17 (i.e., coding isperformed in order of 16 to 1).

The three scan orders proposed in FIG. 17 are selected according to anintra prediction mode. Accordingly, the present disclosure may beconfigured to determine a scan order according to an intra predictionmode identically with respect to L transform coefficients.

FIG. 18 is an embodiment to which the present disclosure is applied, andillustrates the locations of valid transform coefficients and a forwardscan order for each 4×4 block when a diagonal scan is applied and a 4×4RST is applied in a top-left 4×8 block.

If the diagonal scan order of FIG. 17 is followed, a top-left 4×8 blockis divided into 4×4 blocks, and a 4×4 RST is applied to each of the 4×4blocks, assuming that an L value is 8 (i.e., if only 8 of 16 transformcoefficients are left), transform coefficients may be located like FIG.18.

Only half of each 4×4 block may have transform coefficients, and a value0 may be assigned to locations indicated as X by default.

Accordingly, residual coding may be applied assuming that L transformcoefficients are disposed in each 4×4 block according to the scan orderproposed in FIG. 17 and the remaining (16−L) locations of each 4×4 blockare filled with 0.

FIG. 19 is an embodiment to which the present disclosure is applied, andillustrates a case where valid transform coefficients of two 4×4 blocksare merged into a single 4×4 block when a diagonal scan is applied and a4×4 RST is applied in a top-left 4×8 block.

Referring to FIG. 19, L transform coefficients disposed in two 4×4blocks may be merged into one. In particular, if an L value is 8, thetransform coefficients of the two 4×4 blocks fully fill one 4×4 blockand are merged. Accordingly, any transform coefficient is not left inanother 4×4 block.

Accordingly, a corresponding coded_sub_block_flag can be coded as 0because most of residual coding is unnecessary for the empted 4×4blocks.

Furthermore, as an embodiment, various methods of mixing transformcoefficients of two 4×4 blocks may be used. The transform coefficientsmay be merged according to a given order, but an embodiment of thepresent disclosure may provide the following methods.

1) Transform coefficients of two 4×4 blocks are alternately mixed in ascan order. That is, assuming that transform coefficients of an upperblock in FIG. 18 are c₀ ^(u), c₁ ^(u), c₂ ^(u), c₃ ^(u), c₄ ^(u), c₅^(u), c₆ ^(u), and c₇ ^(u), and transform coefficient of a lower blockare c₀ ^(l), c₁ ^(l), c₂ ^(l), c₃ ^(l), c₄ ^(l), c₅ ^(l), c₆ ^(l), andc₇ ^(l), the transform coefficients may be alternately mixed like c₀^(u), c₀ ^(l), c₁ ^(u), c₁ ^(l), c₂ ^(u), c₂ ^(l), . . . c₇ ^(u), and c₇^(l), one by one. Alternatively, order of c_(#) ^(u) and c_(#) ^(l) maybe changed. That is, c_(#) ^(l) may be configured to first emerge.

2) Transform coefficients for the first 4×4 block may be first disposed,and transform coefficients for the second 4×4 block may be thendisposed. That is, the transform coefficients may be connected anddisposed like c₀ ^(u), c₁ ^(u), . . . , c₇ ^(u), c₀ ^(l), c₁ ^(l), . . ., and c₇ ^(l). Alternatively, order of the transform coefficients may bechanged like c₀ ^(l), c₁ ^(l), . . . , c₇ ^(l), c₀ ^(u), c₁ ^(u), . . ., and c₇ ^(u).

Embodiment 1-3: Method of Coding NSST Index for 4×4 RST

If a 4×4 RST is applied as in FIG. 18, No. L+1 to No. 16 transformcoefficients of each 4×4 block may be filled with a value 0 according tothe scan order of the transform coefficients.

Accordingly, if a value not 0 occurs in the No. L+1 to the No. 16locations of two 4×4 blocks, it can be seen that a 4×4 RST is notapplied.

If the 4×4 RST has a structure in which one of a transform set preparedlike the NSST is selected and applied, a transform index (may be namedan NSST index in the present embodiment) indicating which transform willbe applied may be signaled.

In one embodiment, it is assumed that the decoder 200 may be aware of anNSST index through bit stream parsing and bit stream parsing isperformed after residual decoding.

If residual decoding is performed and it is checked that any onenon-zero transform coefficient is present between No. L+1 to No. 16, anNSST index may be configured to be not parsed because a 4×4 RST is notapplied.

Accordingly, the NSST index may be selectively parsed if necessary inorder to reduce a signaling cost.

Assuming that the 4×4 RST is applied to a plurality of 4×4 blocks withina specific region as in FIG. 18 (e.g., the same 4×4 RST may be appliedto all the plurality of 4×4 blocks and different 4×4 RST may be appliedto all the plurality of 4×4 blocks), a 4×4 RST to which all the 4×4blocks are applied may be designated through one NSST index. In thiscase, the same 4×4 RST may be designated, and a 4×4 RST to which each ofall the 4×4 blocks is applied may be designated.

Since the 4×4 RST for all the 4×4 blocks and whether to apply the 4×4RST are determined based on one NSST index, whether a non-zero transformcoefficient is present in No. L+1 to No. 16 locations of all the 4×4blocks may be checked during a residual decoding process. If, as aresult of the check, a non-zero transform coefficient is present in anot-allowed location in one 4×4 block (from the No. L+1 to No. 16locations), an NSST index may not be coded.

The NSST index may be separately signaled with each of a luminance blockand a chrominance block. In the case of a chrominance block, a separateNSST index may be signaled with respect to Cb and Cr. One NSST index maybe shared between a luminance block and a chrominance block.

If one NSST index is shared with respect to Cb and Cr, a 4×4 RSTindicated by the same NSST index may be applied. In this case, the 4×4RSTs for Cb and Cr may be the same, and may have the same NSST index,but an individual 4×4 RST may be used.

If the aforementioned conditional signaling is to be applied to theshared NSST index, whether a non-zero transform coefficient is presentfrom No. L+1 to No. 16 may be checked with respect to all 4×4 blocks forCb and Cr. If the non-zero transform coefficient is present, signalingfor the NSST index may be not signaled.

As in FIG. 19, if transform coefficients for two 4×4 blocks are merged,when a 4×4 RST is applied, after whether a non-zero transformcoefficient is present at a location where a valid transform coefficientis not present is checked, whether to signal an NSST index may bedetermined.

For example, as in FIG. 19b , when an L value is 8 and a 4×4 RST isapplied, if valid transform coefficients for one 4×4 block (blockindicated as X) are not present, coded_sub_block_flag of a block inwhich valid transform coefficients are not present may be checked. Inthis case, if the coded_sub_block_flag is 1, an NSST index may be not besignaled.

Embodiment 1-4: Optimization Method when Coding for NSST Index isPerformed Prior to Residual Coding

If coding for an NSST index is performed prior to residual coding,whether to apply a 4×4 RST is previously determined. Accordingly,residual coding may be omitted with respect to locations to which atransform coefficient is assigned as 0.

In this case, whether to apply the 4×4 RST may be configured to be awarethrough an NSST index. For example, when the NSST index is 0, the 4×4RST is not applied.

Alternatively, whether to apply the 4×4 RST may be signaled through aseparate syntax element (e.g., NSST flag). For example, if the separatesyntax element is an NSST flag, the decoder 200 may check whether toapply the 4×4 RST by first parsing the NSST flag, and may omit residualcoding for locations where a valid transform coefficient cannot bepresent when an NSST flag value is 1.

As an embodiment, the encoder 100 first codes the last non-zerotransform coefficient location on a TU when performing residual coding.If coding for an NSST index is performed after the last non-zerotransform coefficient coding, assuming the application of a 4×4 RST to alocation of the last non-zero transform coefficient, if the location isdetermined as a location where a non-zero transform coefficient may notoccur, the encoder 100 may configure the 4×4 RST to be not appliedwithout coding the NSST index.

For example, in FIG. 18, in the case of locations indicated as X, validtransform coefficients are not located (e.g., a value 0, etc. may befilled) when a 4×4 RST is applied. Accordingly, if the last non-zerotransform coefficient is located in a region indicated as X, the encoder100 may omit coding for an NSST index. If the last non-zero transformcoefficient is not located in the region indicated as X, coding for theNSST index may be performed.

As an embodiment, after coding for the last non-zero transformcoefficient location, if whether to apply the 4×4 RST is checked byconditionally coding the NSST index, the remaining residual codingportion may be processed using the following two methods.

1) If the 4×4 RST is not applied, common residual coding is used withoutany change. That is, coding is performed assuming that any non-zerotransform coefficient may be present in any location from the lastnon-zero transform coefficient location to a location corresponding toDC.

2) If the 4×4 RST is applied, a transform coefficient is not present ina specific location or a specific 4×4 block (e.g., in FIG. 18, an Xlocation may be filled with 0 by default). Residual coding for alocation or block where a transform coefficient is not present may notbe performed.

For example, in FIG. 18, when a location indicated as X is reached,coding for sig_coeff_flag may be omitted. In this case, sig_coeff_flagis an example of a flag indicating whether a non-zero transformcoefficient for a corresponding location is present.

As in FIG. 19, if transform coefficients of two blocks are merged intoone block, coding for coded_sub_block_flag for a 4×4 block assigned as 0may be omitted, and a corresponding value (coded_sub_block_flag) may beset to 0.

If an NSST index is coded after coding for the last non-zero transformcoefficient location, when an x location (P_(x)) and y location (P_(y))of the last non-zero transform coefficient are smaller than an xlocation (T_(x)) and y location (T_(y)) of a transform block,respectively, coding for the NSST index may be omitted, and a 4×4 RSTmay not be applied.

For example, when T_(x)=1, T_(y)=1, this means that an NSST index codingis omitted if the last non-zero transform coefficient is present at a DClocation.

A method of determining whether to code the NSST index through acomparison with such a threshold value may be differently applied toeach of a luminance and a chrominance. For example, T_(x) and T_(y) maybe differently applied to a luminance and a chroma, a threshold valuemay be applied to a luminance and not be applied to a chroma, or viceversa.

The aforementioned two methods, that is, first, a method of omittingcoding for an NSST index when the last non-zero transform coefficient islocated in a region where a valid transform coefficient is not presentand, second, a method of omitting coding for an NSST index when Xcoordinates and Y coordinates for the last non-zero transformcoefficient is smaller than a threshold value, may be applied together.

For example, after a threshold value of a location coordinate of thelast non-zero transform coefficient is first checked, whether the lastnon-zero transform coefficient is present in a region where a validtransform coefficient is not present may be checked, or order may bechanged.

The methods proposed in the present embodiment 4 may also be applied toan 8×8 RST. That is, if the last non-zero transform coefficient islocated in a region other than a top-left 4×4 within a top-left 8×8region, coding for an NSST index may be omitted, and coding for an NSSTindex may be performed, if not.

Furthermore, if both X and Y coordinates values for the last non-zerotransform coefficient location are less than a threshold value, codingfor an NSST index may be omitted. Alternatively, the two methods may beapplied together.

Embodiment 1-5: Different NSST Index Coding and Residual Coding Methodsare Applied to Luminance Block and Chrominance Block when RST is Applied

The methods described in Embodiment 1-3 and Embodiment 1-4 may bedifferently applied to a luminance and a chrominance. That is, codingfor an NSST index and residual coding methods for a luminance and achrominance may be differently applied.

For example, the method of Embodiment 1-4 may be applied to a luminanceblock, and the method of Embodiment 1-3 may be applied to a chrominanceblock. Alternatively, the conditional NSST index coding proposed inEmbodiment 1-3 or Embodiment 1-4 may be applied to a luminance block,and conditional NSST index coding may not be applied to a chrominanceblock, or vice versa.

FIG. 20 is an embodiment to which the present disclosure is applied, andillustrates a flowchart in which a video signal is encoded based on areduced secondary transform (RST).

The encoder 100 may determine (or select) a forward secondary transformbased on at least one of a prediction mode, block shape and/or blocksize of a current block (S2010). In this case, a candidate for theforward secondary transform may include at least one of the embodimentsof FIGS. 6 and/or 12.

The encoder 100 may determine an optimal forward secondary transformthrough rate-distortion (RD) optimization. The optimal forward secondarytransform may correspond to one of a plurality of transformcombinations. The plurality of transform combinations may be defined bya transform index. For example, for the RD optimization, the encoder 100may compare the results of execution of all of a forward secondarytransform, quantization, and residual coding for each of the candidates.In this case, an equation, such as cost=rate+λ·distortion orcost=distortion+λ·rate, may be used, but the present disclosure is notlimited thereto.

The encoder 100 may signal a secondary transform index corresponding tothe optimal forward secondary transform (S2020). In this case, otherembodiments described in the present disclosure may be applied to thesecondary transform index.

For example, the transform set configuration of FIG. 12 may be used forthe secondary transform index. One transform set is composed of two orthree transforms according to an intra prediction mode, and may beconfigured to select one of a maximum of four transforms including acase where the secondary transform is not applied. Assuming that indices0, 1, 2, and 3 are assigned to four transforms, respectively, theencoder 100 may designate a transform to be applied by signaling asecondary transform index to each transform coefficient block. In thiscase, the encoder 100 may assign the No. 0 index as an identity matrix,that is, a case where the secondary transform is not applied.

As another embodiment, the signaling of the secondary transform indexmay be performed in any one step of 1) prior to residual coding, 2) inthe middle of the residual coding (after the last non-zero transformcoefficient coding), or 3) after the residual coding. The aboveembodiments are described in detail as follows.

1) A method of signaling a secondary transform index prior to residualcoding

The encoder 100 may determine a forward secondary transform.

The encoder 100 may code a secondary transform index corresponding tothe forward secondary transform.

The encoder 100 may code the location of the last non-zero transformcoefficient.

The encoder 100 may perform residual coding on syntax elements otherthan the location of the last non-zero transform coefficient.

2) A method of signaling a secondary transform index in the middle ofresidual coding

The encoder 100 may determine a forward secondary transform.

The encoder 100 may code the location of the last non-zero transformcoefficient.

If the last non-zero transform coefficient is not located in a specificregion, the encoder 100 may code a secondary transform indexcorresponding to a forward secondary transform. In this case, if areduced secondary transform is applied, the specific region indicatesthe remaining regions other than a location wherein a non-zero transformcoefficient may be present when transform coefficients are disposedaccording to a scan order. However, the present disclosure is notlimited thereto.

The encoder 100 may perform residual coding on the syntax elements otherthan the location of the last non-zero transform coefficient.

3) A method of signaling a secondary transform index after residualcoding

The encoder 100 may determine a forward secondary transform.

The encoder 100 may code the location of the last non-zero transformcoefficient.

If the last non-zero transform coefficient is not located in a specificregion, the encoder 100 may perform residual coding on syntax elementsother than the location of the last non-zero transform coefficient. Inthis case, if a reduced secondary transform is applied, the specificregion indicates the remaining regions other than a location where anon-zero transform coefficient may be present when transformcoefficients are disposed according to a scan order. However, thepresent disclosure is not limited thereto.

The encoder 100 may code a secondary transform index corresponding tothe forward secondary transform.

Meanwhile, the encoder 100 may perform a forward primary transform onthe current block (residual block) (S2030). In this case, steps S2010and/or S2020 may be similarly applied to the forward primary transform.

The encoder 100 may perform a forward secondary transform on the currentblock using an optimal forward secondary transform (S2040). For example,the optimal forward secondary transform may be a reduced secondarytransform. The reduced secondary transform means that a transform bywhich L (L<N) transform coefficient data (L×1 transform coefficientvector) is output when N residual data (N×1 residual vector) is input.

As an embodiment, the reduced secondary transform may be applied to aspecific region of the current block. For example, when the currentblock is N×N, the specific region may mean a top-left N/2×N/2 region.However, the present disclosure is not limited thereto, and bedifferently configured based on at least one of a prediction mode, ablock shape, or a block size. For example, when the current block isN×N, the specific region may mean a top-left M×M region (M<=N).

Meanwhile, the encoder 100 may generate a transform coefficient block byperforming quantization on the current block (S2050).

The encoder 100 may generate a bit stream by performing entropy encodingon the transform coefficient block.

FIG. 21 is an embodiment to which the present disclosure is applied, andillustrates a flowchart in which a video signal is decoded based on areduced secondary transform (RST).

The decoder 200 may obtain a secondary transform index from a bit stream(S2110). In this case, other embodiments described in the presentdisclosure may be applied to the secondary transform index. For example,the secondary transform index may include at least one of theembodiments of FIGS. 6A, 6B and/or 12.

As another embodiment, the step of obtaining the secondary transformindex may be performed in any one step of 1) before residual decoding,2) in the middle of the residual decoding (after the decoding of thelast non-zero transform coefficient), or 3) after the residual decoding.

The decoder 200 may derive a secondary transform corresponding to thesecondary transform index (S2120). In this case, a candidate for thesecondary transform may include at least one of the embodiments of FIGS.6 and/or 12.

However, steps S2110 and S2120 are embodiments, and the presentdisclosure is not limited thereto. For example, the decoder 200 mayderive the secondary transform based on at least one of a predictionmode, block shape and/or block size of a current block without obtainingthe secondary transform index.

Meanwhile, the decoder 200 may obtain a transform coefficient block byentropy-decoding the bit stream, and may perform inverse quantization onthe transform coefficient block (S2130).

The decoder 200 may perform an inverse secondary transform on theinverse-quantized transform coefficient block (S2140). For example, theinverse secondary transform may be a reduced secondary transform. Thereduced secondary transform means a transform by which L (L<N) transformcoefficient data (L×1 transform coefficient vector) is output when Nresidual data (N×1 residual vector) is input.

As an embodiment, the reduced secondary transform may be applied to aspecific region of a current block. For example, when the current blockis N×N, the specific region may mean a top-left N/2×N/2 region. However,the present disclosure is not limited thereto, the specific region maybe differently configured based on at least one of prediction mode, ablock shape, or a block size. For example, when the current block isN×N, the specific region may mean a top-left M×M region (M<=N) or M×L(M<=N, L<=N).

Furthermore, the decoder 200 may perform an inverse primary transform onthe results of the inverse secondary transform (S2150).

The decoder 200 may generate a residual block through step S2150, andgenerates a reconstructed by adding the residual block and a predictionblock.

A process of improving coding efficiency by applying an RST to a 4×4block has been described through the aforementioned embodiments 1-1 to1-5. In the following embodiments, a method and apparatus for reducingcomputational complexity by omitting a secondary transform for atransform block having a 4×4 size is described.

Embodiment 2-1: Non-Application of Secondary Transform to 4×4 TransformBlock

FIG. 22 illustrates an example of a transform block and a top-left 4×4region according to an embodiment of the present disclosure.

As illustrated in FIG. 22, a secondary transform may be configured to beapplied to only a top-left 4×4 region of a transform block (transformunit or TU) having width×height (width>=4, height>=4).

In this case, if a non-separable transform is applied to the top-left4×4 region as the secondary transform, a heavy computational load mayoccur compared to a case where a separable transform is applied. Forexample, if the non-separable transform is given as a 16×16 matrix, whenthe secondary transform is applied to the top-left 4×4 region, a totalof 256 multiplications are necessary.

In the worse case, if one picture is divided into 4×4 transform blocks,when a secondary transform is applied to each of the 4×4 transformblocks (in the case of a secondary transform having a matrix form), 256multiplications may be necessary. Accordingly, a great computationalload is necessary compared to a primary transform. For example, if aseparable transform is applied to a 4×4 transform block as a primarytransform in a matrix multiplication form, a total of 2×4³=128multiplications are necessary. If a primary transform and a secondarytransform are configured as respective pipeline steps and the primarytransform and the secondary transform are performed in a pipeliningform, a computational load for the secondary transform is much great.Accordingly, overall performance may be degraded because the secondarytransform becomes a critical path in pipelining.

Accordingly, the secondary transform may be forced to be not applied toa case where the size of a transform block is 4×4. In other words, ifthe size of a transform block is 4×4, the secondary transform may beforced to be applied up to only a primary transform.

Embodiment 2-2: Selection of a Secondary Transform in Embodiment 2-1

In Embodiment 2-1, there is proposed a method of applying a secondarytransform to only a top-left 4×4 region of a transform block. Examplesof the secondary transforms which may be applied in the configuration ofEmbodiment 2-1 are as follows.

1) A reduced secondary transform (RST) which may be applied to a 4×4region: a matrix of an L×16 (L<=16) form (corresponding to a forwardtransform).

2) A sparse orthonormal transform (SOT) or KLT which may be applied to a4×4 region: it may be considered to correspond to the case of L=16 in1), and may be considered as a transform having a 16×16 matrix form froma computational load viewpoint.

3) AJEM NSST which may be applied to a 4×4 region: this is a Givensrotation-based transform having 16 data inputs and 16 data outputs andhas been described with reference to FIGS. 13 to 15.

4) A layered Givens transform (LGT) which may be applied to a 4×4region: this is a Givens rotation-based transform having 16 data inputsand 16 data outputs as in 3), but is characterized in that a designatedpermutation is applied between two adjacent Givens rotation layers.

Basically, in addition to the aforementioned secondary transform, othertypes of secondary transforms may be applied.

Embodiment 2-3: Selection of a Primary Transform when a Transform Blockis 4×4 in the Configuration of Embodiment 2-1

In the configuration of Embodiment 2-1, a secondary transform is notapplied to a case where a transform block is 4×4. Accordingly, a primarytransform may be configured to be applied to a case where a transformblock is 4×4. In this case, the primary transform which may be appliedis listed as follows.

1) As an MTS (EMT or AMT) combination to which JEM is applied, thetransforms described with reference to FIGS. 6a and 6b are applied

2) DCT-2 is applied to both a horizontal transform and a verticaltransform

3) DST-7 or DCT-8 may be applied in a horizontal direction, and DST-7 orDCT-8 may be applied in a vertical direction. Transform pairs may becomposed of transform pairs which may be applied to residual samplesgenerated through an intra prediction and a transform pair which may beapplied to residual samples generated through an inter prediction,respectively, as in FIGS. 23a and 23 b.

4) DST-7 is applied as both a horizontal transform and a verticaltransform

FIG. 23 illustrates an example of a table for determining a transformtype in a horizontal direction and a vertical direction according to aprediction mode, wherein FIG. 23a illustrates a transform pair to whicha residual generated through an intra prediction is applied, and FIG.23b illustrates a transform pair to which a residual generated throughan inter prediction is applied.

Referring to FIG. 23a , if a transform is applied to a residual signalgenerated through an intra prediction or an inverse transform is appliedto an inverse-quantized or inverse secondary-transformed signal, anindex indicative of a transform type may be determined both in thehorizontal direction and the vertical direction. For example, if an MTSindex is 0, DST-7 is determined as a transform type for the horizontaldirection and the vertical direction. When the MTS index is 1, DCT-8 isdetermined as a transform type for the horizontal direction, and DST-7is determined as a transform type for the vertical direction. When theMTS index is 2, DST-7 is determined as a transform type for thehorizontal direction, and DCT-8 is determined as a transform type forthe vertical direction. When the MTS index is 3, DCT-8 is determined asa transform type for both the horizontal direction and the verticaldirection.

Referring to FIG. 23b , if a transform is applied to a residual signalgenerated through an inter prediction or an inverse transform is appliedto an inverse-quantized or inverse secondary-transformed signal, atransform type corresponding to an index indicative of a transform typemay be determined in each of a horizontal direction and a verticaldirection. For example, when an MTS index is 0, DCT-8 is determined as atransform type for both the horizontal direction and the verticaldirection. When the MTS index is 1, DST-7 is determined as a transformtype for the horizontal direction, and DCT-8 is determined as atransform type for the vertical direction. When the MTS index is 2,DCT-8 is determined as a transform type for the horizontal direction,and DST-7 is determined as a transform type for the vertical direction.When the MTS index is 3, DST-7 is determined as a transform type forboth the horizontal direction and the vertical direction.

Embodiment 2-4: Application of the Secondary Transform of Embodiment 2-2and the Primary Transform of Embodiment 2-3 in the Configuration ofEmbodiment 2-1

In the configuration of Embodiment 2-1, one of the secondary transformsproposed in Embodiment 2-2 may be applied to a top-left 4×4, and one ofthe primary transforms proposed in Embodiment 2-3 may be applied to a4×4 transform block. A primary transform for a case where the size of atransform block is not 4×4 may be the same as or different from aprimary transform applied to a 4×4 transform block. The tablesillustrated in FIGS. 23a and 23b may be used for the JEM MTS illustratedin FIG. 6a and 6b or a transform pair of the JEM MTS as a primarytransform for a case where the size of a transform block is not 4×4.

FIG. 24 illustrates a flowchart of a method of encoding a video signalaccording to an embodiment of the present disclosure. FIG. 24illustrates an example of an operation of the encoder 100 of FIG. 1.

At step S2410, the encoder 100 may check a transform block includingresidual samples after a prediction. The encoder 100 may deriveprediction samples through a prediction for a current block. Forexample, the encoder 100 may determine whether to perform an interprediction or whether to perform an intra prediction on the currentblock, and may determine a detailed prediction mode (detailed intraprediction mode or detailed intra prediction mode) based on arate-distortion (RD) cost. The encoder 100 may derive the predictionsamples for the current block based on the determined mode. Thereafter,the encoder 100 may derive the residual samples by comparing originalsamples in the current block with the prediction samples. For example,the encoder 100 may derive the residual samples by removing theprediction samples from the current block. Furthermore, the encoder 100may check a transform block composed of the residual samples.

At step S2420, the encoder 100 may generate transform coefficientsthrough a transform for the residual samples included in the transformblock. More specifically, the encoder 100 may generate transformcoefficients of a frequency domain by applying at least one transform toresidual samples of a space domain included in the transform block. Adetailed transform process according to an embodiment of the presentdisclosure is described in detail with reference to FIG. 25.

At step S2430, the encoder 100 may perform quantization and entropycoding on the transform coefficients. More specifically, the encoder 100may generate an encoded video signal by performing quantization andentropy coding on the transform samples generated at step S2420. Forexample, the encoder 100 may encode image information, includinginformation on the prediction and information on the residual samples,and may output the encoded image information in a bit stream form. Inthis case, the information on a prediction may include prediction modeinformation and motion information (if an inter prediction is applied)as information related to a prediction procedure. The information on theresidual samples may include information on the quantized transformcoefficients. An output bit stream may be delivered to the decoder 200through a storage medium or over a network.

In other words, the encoder 100 may derive prediction samples through aprediction for a current block, may generate residual samples for thecurrent block based on the prediction samples, may derive transformcoefficients through a transform (primary transform/secondary transform)procedure, may derive quantized transform coefficients throughquantization, and may encode image information including predictioninformation and residual information.

FIG. 25 illustrates an example of a transform process according to thesize of a transform block in an encoding process of a video signalaccording to an embodiment of the present disclosure. FIG. 25illustrates an example of step S2420 of FIG. 24.

At step S2510, the encoder 100 may perform a forward primary transformon a transform block including residual samples. More specifically, theencoder 100 may apply the forward primary transform to the transformblock including the residual samples in each of a horizontal directionand a vertical direction. In this case, the forward primary transformmay be denoted as a separable primary transform. According to anembodiment of the present disclosure, the forward primary transform mayinclude a horizontal transform and a vertical transform. Furthermore,the forward primary transform may be determined by a combination ofDCT-2, DST-7, or DCT-8. For example, the forward primary transform maybe configured like FIGS. 6a and 6b , or FIGS. 24a and 24 b.

At step S2520, the encoder 100 may determine whether the size of atransform block is smaller than or equal to 4×4. In this case, the sizeof the transform block may mean the number of samples included in thetransform block. The size 4×4 of the transform block indicates that thenumber of samples in each of the horizontal direction and the verticaldirection is four, and means that the number of horizontal direction(row direction) components is four and the number of vertical direction(column direction) components is four. In one embodiment, the transformblock may be configured so that a horizontal direction (row direction)size and a vertical direction (column direction) size are the same. Inthis case, whether the transform horizontal direction size or verticaldirection size is smaller than or equal to 4 may be checked. In anotherembodiment, the transform block may be configured so that the horizontaldirection (row direction) size and the vertical direction (columndirection) size are different. In this case, whether a total number ofsamples included in the transform block is smaller than or equal to4×4=16 may be checked.

When the size of the transform block is smaller than or equal to 4×4,the encoder 100 may proceed to step S2530. At step S2530, the encoder100 may omit a forward secondary transform. That is, when the size ofthe transform block is smaller than or equal to 4×4, the encoder 100 mayperform quantization on transform coefficients in the state in which anadditional transform has not been performed on transform coefficientsgenerated by applying the forward primary transform.

When the size of the transform block is greater than 4×4, the encoder100 may proceed to step S2540. At step S2540, the encoder 100 may applya forward secondary transform. In other words, the encoder 100 maygenerate modified transform coefficients by additionally applying asecondary transform on the transform coefficients generated as theresults of the forward primary transform. In this case, a non-separablesecondary transform may be used as the forward secondary transform.According to an embodiment of the present disclosure, the forwardsecondary transform may be applied to a top-left 4×4 region of thetransform block. In this case, the non-separable secondary transform mayinclude at least one of a reduced secondary transform (RST) having amatrix of an L×16 form, wherein L is smaller than or equal to 16, asparse orthonormal transform (SOT) having a matrix of a 16×16 form, aGivens rotation-based transform having 16 inputs and 16 outputs, or alayered givens transform having 16 inputs and 16 outputs and applyingdesignated permutation between two neighboring Givens rotation layers.

FIG. 26 illustrates a flowchart of a process of decoding a video signalaccording to an embodiment of the present disclosure. FIG. 26illustrates an example of an operation of the decoder 100 of FIG. 2.

At step S2610, the decoder 100 may generate a transform block includingtransform coefficients through entropy decoding and inversequantization. The decoder 100 may generate the transform block composedof the transform coefficients, that is, a target of an inversetransform, through entropy decoding and inverse quantization for a bitstream of a video signal to be decoded.

At step S2620, the decoder 100 may generate residual samples through aninverse transform for the transform coefficients of the transform block.An inverse transform process according to an embodiment of the presentdisclosure is described in detail with reference to FIG. 27.

At step S2630, the decoder 100 may generate a reconstructed picture fromthe residual samples through a prediction. The decoder 100 may deriveprediction samples by performing an inter prediction or an intraprediction on a current block based on prediction information. Thedecoder 100 may generate an original picture by merging the residualsamples generated through the inverse transform process and theprediction samples generated through the prediction process.

FIG. 27 illustrates an example of a transform process according to thesize of a transform block in a process of decoding a video signalaccording to an embodiment of the present disclosure.

At step S2710, the decoder 200 may determine whether the size of atransform block is smaller than or equal to 4×4. In this case, the sizeof the transform block may mean the number of transform coefficientsincluded in the transform block. The size of the transform block 4×4indicates that the number of samples in each of a horizontal directionand a vertical direction is four, and means that the number ofhorizontal direction (row direction) components is four and the numberof vertical direction (column direction) components is four. In oneembodiment, the transform block may be configured to be the same as ahorizontal direction (row direction) size and a vertical direction(column direction) size. In this case, whether the transform horizontaldirection size or the vertical direction size is smaller than or equalto 4 may be checked. In another embodiment, the transform block may beconfigured so that the horizontal direction (row direction) size and thevertical direction (column direction) size are different. In this case,whether a total number of coefficients included in the transform blockis smaller than or equal to 4×4=16 may be checked.

When the size of the transform block is smaller than or equal to 4×4,the decoder 100 may proceed to step S2720. At step S2720, the decoder200 may omit an inverse secondary transform. That is, when the size ofthe transform block is smaller than or equal to 4×4, the decoder 100 mayproceed to step S2740 without an inverse secondary transform, may applyan inverse primary transform, and may perform a prediction on residualsamples generated as the results of the inverse primary transform.

When the size of the transform block is greater than 4×4, the decoder200 may proceed to step S2730. At step S2730, the decoder 200 may applyan inverse secondary transform. In this case, a non-separable secondarytransform may be used as the inverse secondary transform. According toan embodiment of the present disclosure, the inverse secondary transformmay be applied to a top-left 4×4 region of the transform block. In thiscase, the non-separable secondary transform may include at least one ofa reduced secondary transform (RST) having a matrix of an L×16 form,wherein L is smaller than or equal to 16, a sparse orthonormal transform(SOT) having a matrix of a 16×16 form, a Givens rotation-based transformhaving 16 inputs and 16 outputs, or a layered givens transform having 16inputs and 16 outputs and applying designated permutation between twoneighboring Givens rotation layers. The decoder 200 may generate atransform block including modified transform coefficients as the resultsof the inverse secondary transform.

After the inverse secondary transform is omitted as in step S2720 or theinverse secondary transform is applied as in step S2730, the decoder 200may proceed to step S2710. At step S2710, the decoder 200 may perform aninverse primary transform on the transform coefficients, generated byapplying inverse quantization, or the transform block including modifiedtransform coefficients generated by applying the inverse secondarytransform. More specifically, the decoder 200 may apply an inverseprimary transform to the transform block including transformcoefficients in each of a horizontal direction and a vertical direction.In this case, the inverse primary transform may be denoted as aseparable primary transform. According to an embodiment of the presentdisclosure, the inverse primary transform may include a horizontaltransform and a vertical transform. Furthermore, the inverse primarytransform may be determined based on DCT-2, DST-7, or DCT-8. Forexample, the inverse primary transform may be configured like FIGS. 6aand 6b or FIGS. 24a and 24 b.

A secondary transform which may be applied to the present disclosure isnot limited to the aforementioned transforms. Various types ofnon-separable secondary transforms which may be applied between theforward primary transform step and the quantization step based on theencoder and between the inverse quantization step and the inverseprimary transform step based on the decoder may be applied to thepresent disclosure.

FIG. 28 illustrates an example of a video coding system as an embodimentto which the present disclosure is applied.

A video coding system may include a source device and a receivingdevice. The source device may transmit encoded video/image informationor data to the receiving device via a digital storage medium or anetwork in a file or streaming form.

The source device may include a video source, an encoding apparatus, anda transmitter. The receiving device may include a receiver, a decodingapparatus, and a renderer. The encoding apparatus may be called avideo/image encoding apparatus, and the decoding apparatus may be calleda video/image decoding apparatus. The transmitter may be included in theencoding apparatus. The receiver may be included in the decodingapparatus. The renderer may include a display, and the display may beimplemented as a separate device or an external component.

The video source may acquire a video/image through a capturing,synthesizing, or generating process of the video/image. The video sourcemay include a video/image capture device and/or a video/image generationdevice. The video/image capture device may include, for example, one ormore cameras, video/image archives including previously capturedvideo/images, and the like. The video/image generation device mayinclude, for example, a computer, a tablet, and a smart phone and may(electronically) generate the video/image. For example, a virtualvideo/image may be generated by the computer, etc., and in this case,the video/image capturing process may be replaced by a process ofgenerating related data.

The encoding apparatus may encode an input video/image. The encodingapparatus may perform a series of procedures including prediction,transform, quantization, and the like for compression and codingefficiency. The encoded data (encoded video/image information) may beoutput in the bitstream form.

The transmitter may transfer the encoded video/image information or dataoutput in the bitstream to the receiver of the receiving device throughthe digital storage medium or network in the file or streaming form. Thedigital storage medium may include various storage media such as USB,SD, CD, DVD, Blu-ray, HDD, SSD, and the like. The transmitter mayinclude an element for generating a media file through a predeterminedfile format and may include an element for transmission through abroadcast/communication network. The receiver may extract the bitstreamand transfer the extracted bitstream to the decoding apparatus.

The decoding apparatus may perform a series of procedures includingdequantization, inverse transform, prediction, etc., corresponding to anoperation of the encoding apparatus to decode the video/image.

The renderer may render the decoded video/image. The renderedvideo/image may be displayed by the display.

FIG. 29 illustrates an example of a video streaming system as anembodiment to which the present disclosure is applied.

Referring to FIG. 29, the content streaming system to which the presentdisclosure is applied may basically include an encoding server, astreaming server, a web server, a media storage, a user equipment and amultimedia input device.

The encoding server basically functions to generate a bitstream bycompressing content input from multimedia input devices, such as asmartphone, a camera or a camcorder, into digital data, and to transmitthe bitstream to the streaming server. For another example, ifmultimedia input devices, such as a smartphone, a camera or a camcorder,directly generate a bitstream, the encoding server may be omitted.

The bitstream may be generated by an encoding method or bitstreamgeneration method to which the present disclosure is applied. Thestreaming server may temporally store a bitstream in a process oftransmitting or receiving the bitstream.

The streaming server transmits multimedia data to the user equipmentbased on a user request through the web server. The web server plays arole as a medium to notify a user that which service is provided. When auser requests a desired service from the web server, the web servertransmits the request to the streaming server. The streaming servertransmits multimedia data to the user. In this case, the contentstreaming system may include a separate control server. In this case,the control server functions to control an instruction/response betweenthe apparatuses within the content streaming system.

The streaming server may receive content from the media storage and/orthe encoding server. For example, if content is received from theencoding server, the streaming server may receive the content in realtime. In this case, in order to provide smooth streaming service, thestreaming server may store a bitstream for a given time.

Examples of the user equipment may include a mobile phone, a smartphone, a laptop computer, a terminal for digital broadcasting, personaldigital assistants (PDA), a portable multimedia player (PMP), anavigator, a slate PC, a tablet PC, an ultrabook, a wearable device(e.g., a watch type terminal (smartwatch), a glass type terminal (smartglass), and a head mounted display (HMD)), digital TV, a desktopcomputer, and a digital signage.

The servers within the content streaming system may operate asdistributed servers. In this case, data received from the servers may bedistributed and processed.

The embodiments described in the present disclosure may be implementedand performed on a processor, a microprocessor, a controller, or a chip.For example, functional units illustrated in each drawing may beimplemented and performed on a computer, the processor, themicroprocessor, the controller, or the chip.

In addition, the decoder and the encoder to which the present disclosuremay be included in a multimedia broadcasting transmitting and receivingdevice, a mobile communication terminal, a home cinema video device, adigital cinema video device, a surveillance camera, a video chat device,a real time communication device such as video communication, a mobilestreaming device, storage media, a camcorder, a video on demand (VoD)service providing device, an OTT (Over the top) video device, anInternet streaming service providing devices, a three-dimensional (3D)video device, a video telephone video device, a transportation meansterminal (e.g., a vehicle terminal, an airplane terminal, a shipterminal, etc.), and a medical video device, etc., and may be used toprocess a video signal or a data signal. For example, the OTT videodevice may include a game console, a Blu-ray player, an Internet accessTV, a home theater system, a smartphone, a tablet PC, a digital videorecorder (DVR), and the like.

In addition, a processing method to which the present disclosure isapplied may be produced in the form of a program executed by thecomputer, and may be stored in a computer-readable recording medium.Multimedia data having a data structure according to the presentdisclosure may also be stored in the computer-readable recording medium.The computer-readable recording medium includes all types of storagedevices and distribution storage devices storing computer-readable data.The computer-readable recording medium may include, for example, aBlu-ray disc (BD), a universal serial bus (USB), a ROM, a PROM, anEPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, andan optical data storage device. Further, the computer-readable recordingmedium includes media implemented in the form of a carrier wave (e.g.,transmission over the Internet). Further, the bitstream generated by theencoding method may be stored in the computer-readable recording mediumor transmitted through a wired/wireless communication network.

In addition, the embodiment of the present disclosure may be implementedas a computer program product by a program code, which may be performedon the computer by the embodiment of the present disclosure. The programcode may be stored on a computer-readable carrier.

As described above, the embodiments described in the present disclosuremay be implemented and performed on a processor, a microprocessor, acontroller or a chip. For example, the function units illustrated in thedrawings may be implemented and performed on a computer, a processor, amicroprocessor, a controller or a chip.

Furthermore, the decoder and the encoder to which the present disclosureis applied may be included in a multimedia broadcasting transmission andreception device, a mobile communication terminal, a home cinema videodevice, a digital cinema video device, a camera for monitoring, a videodialogue device, a real-time communication device such as videocommunication, a mobile streaming device, a storage medium, a camcorder,a video on-demand (VoD) service provision device, an over the top (OTT)video device, an Internet streaming service provision device, athree-dimensional (3D) video device, a video telephony device, and amedical video device, and may be used to process a video signal or adata signal. For example, the OTT video device may include a gameconsole, a Blu-ray player, Internet access TV, a home theater system, asmartphone, a tablet PC, and a digital video recorder (DVR).

Furthermore, the processing method to which the present disclosure isapplied may be produced in the form of a program executed by a computer,and may be stored in a computer-readable recording medium. Multimediadata having a data structure according to the present disclosure mayalso be stored in a computer-readable recording medium. Thecomputer-readable recording medium includes all types of storage devicesin which computer-readable data is stored. The computer-readablerecording medium may include a Blu-ray disk (BD), a universal serial bus(USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetictape, a floppy disk, and an optical data storage device, for example.Furthermore, the computer-readable recording medium includes mediaimplemented in the form of carriers (e.g., transmission through theInternet). Furthermore, a bit stream generated using an encoding methodmay be stored in a computer-readable recording medium or may betransmitted over wired and wireless communication networks.

Furthermore, an embodiment of the present disclosure may be implementedas a computer program product using program code. The program code maybe performed by a computer according to an embodiment of the presentdisclosure. The program code may be stored on a carrier readable by acomputer.

The embodiments described above are implemented by combinations ofcomponents and features of the present disclosure in predeterminedforms. Each component or feature should be considered selectively unlessspecified separately. Each component or feature may be carried outwithout being combined with another component or feature. Moreover, somecomponents and/or features are combined with each other and canimplement embodiments of the present disclosure. The order of operationsdescribed in embodiments of the present disclosure may be changed. Somecomponents or features of one embodiment may be included in anotherembodiment, or may be replaced by corresponding components or featuresof another embodiment. It is apparent that some claims referring tospecific claims may be combined with another claims referring to theclaims other than the specific claims to constitute an embodiment or addnew claims by means of amendment after the application is filed.

Embodiments of the present disclosure can be implemented by variousmeans, for example, hardware, firmware, software, or combinationsthereof. When embodiments are implemented by hardware, one embodiment ofthe present disclosure can be implemented by one or more applicationspecific integrated circuits (ASICs), digital signal processors (DSPs),digital signal processing devices (DSPDs), programmable logic devices(PLDs), field programmable gate arrays (FPGAs), processors, controllers,microcontrollers, microprocessors, and the like.

When embodiments are implemented by firmware or software, one embodimentof the present disclosure can be implemented by modules, procedures,functions, etc. performing functions or operations described above.Software code can be stored in a memory and can be driven by aprocessor. The memory is provided inside or outside the processor andcan exchange data with the processor by various well-known means.

It is apparent to those skilled in the art that the present disclosurecan be embodied in other specific forms without departing from essentialfeatures of the present disclosure. Accordingly, the aforementioneddetailed description should not be construed as limiting in all aspectsand should be considered as illustrative. The scope of the presentdisclosure should be determined by rational interpretation of theappended claims, and all modifications within an equivalent scope of thepresent disclosure are included in the scope of the present disclosure.

INDUSTRIAL APPLICABILITY

The aforementioned preferred embodiments of the present disclosure havebeen disclosed for illustrative purposes, and those skilled in the artmay improve, change, substitute, or add various other embodimentswithout departing from the technical spirit and scope of the presentdisclosure disclosed in the attached claims.

1. A method of encoding a video signal, comprising: checking a transformblock including residual samples except a prediction sample from apicture of the video signal; generating transform coefficients through atransform for the residual samples of the transform block based on asize of the transform block; and performing quantization and entropycoding the transform coefficients, wherein generating the transformcoefficients includes: applying a forward primary transform to each of ahorizontal direction and vertical direction of the transform blockincluding the residual samples; and not applying a forward non-separablesecondary transform to the transform block to which the primarytransform has been applied when the size of the transform block issmaller than or equal to 4×4, and applying the forward non-separablesecondary transform to the transform block when the size of thetransform block is greater than 4×4.
 2. The method of claim 1, whereinapplying the forward non-separable secondary transform to the transformblock includes applying the forward non-separable secondary transform toa left-top 4×4 region of the transform block.
 3. The method of claim 1,wherein the forward non-separable secondary transform includes at leastone of a low frequency non-separable transform (LFNST) having a matrixof an L×16 form, wherein the LFNST indicates a non-separable transformapplied to a left-top specific region and L is smaller than or equal to16, a sparse orthonormal transform (SOT) having a matrix of a 16×16form, a transform having 16 inputs and 16 outputs and based on Givensrotation, or a layered givens transform having 16 inputs and 16 outputsand applying a designated permutation between two neighboring Givensrotation layers.
 4. The method of claim 1, wherein the forward primarytransform includes a horizontal transform and a vertical transform forthe transform block, and wherein each of the horizontal transform andthe vertical transform is configured as DCT-2 or configured by acombination of DST-7 and DCT-8.
 5. A method of decoding a video signal,comprising: generating a transform block including transformcoefficients by performing entropy decoding and inverse quantization onthe video signal; generating residual signals through an inversetransform for the transform coefficients of the transform block based ona size of the transform block; and generating a reconstructed picture bygenerating a prediction signal through prediction from the residualsignals, wherein performing the inverse quantization includes: notapplying an inverse non-separable secondary transform to the transformblock when the size of the transform block is smaller than or equal to4×4 and applying the inverse non-separable secondary transform to thetransform block when the size of the transform block is greater than4×4; and applying an inverse primary transform to the transform blockseparately in a horizontal direction and a vertical direction.
 6. Themethod of claim 5, wherein applying the inverse non-separable secondarytransform to the transform block includes applying the inversenon-separable secondary transform to a left-top 4×4 region of thetransform block.
 7. The method of claim 5, wherein the inversenon-separable secondary transform includes at least one of a lowfrequency non-separable transform (LFNST) having a matrix of an L×16form, wherein the LFNST indicates a non-separable transform applied to aleft-top specific region and L is smaller than or equal to 16, a sparseorthonormal transform (SOT) having a matrix of a 16×16 form, a transformhaving 16 inputs and 16 outputs and based on Givens rotation, or alayered givens transform having 16 inputs and 16 outputs and applying adesignated permutation between two neighboring Givens rotation layers,wherein the forward primary transform includes a horizontal transformand a vertical transform for the transform block, and wherein each ofthe horizontal transform and the vertical transform is configured asDCT-2 or configured by a combination of DST-7 and DCT-8.
 8. A videosignal processing apparatus for encoding a video signal, comprising: amemory storing the video signal, and an encoder functionally coupled tothe memory and configured to encoding-process the video signal, whereinthe encoder is configured to: check a transform block including residualsamples except a prediction sample from a picture of the video signal;generate transform coefficients through a transform for the residualsamples of the transform block based on a size of the transform block;and perform quantization and entropy coding the transform coefficients,wherein when generating the transform coefficients through the transformfor the residual samples of the transform block, the encoder isconfigured to: apply a forward primary transform to each of a horizontaldirection and vertical direction of the transform block including theresidual samples, not apply a forward non-separable secondary transformto the transform block to which the primary transform has been appliedwhen the size of the transform block is smaller than or equal to 4×4,and apply the forward non-separable secondary transform to the transformblock when the size of the transform block is greater than 4×4.
 9. Thevideo signal processing apparatus of claim 8, wherein the encoder isconfigured to apply the forward non-separable secondary transform to aleft-top 4×4 region of the transform block when the size of thetransform block is greater than 4×4.
 10. The video signal processingapparatus of claim 8, wherein the forward non-separable secondarytransform includes at least one of a low frequency non-separabletransform (LFNST) having a matrix of an L×16 form, wherein the LFNSTindicates a non-separable transform applied to a left-top specificregion and L is smaller than or equal to 16, a sparse orthonormaltransform (SOT) having a matrix of a 16×16 form, a transform having 16inputs and 16 outputs and based on Givens rotation, or a layered givenstransform having 16 inputs and 16 outputs and applying a designatedpermutation between two neighboring Givens rotation layers.
 11. Thevideo signal processing apparatus of claim 8, wherein the forwardprimary transform includes a horizontal transform and a verticaltransform for the transform block, and wherein each of the horizontaltransform and the vertical transform is configured as DCT-2 orconfigured by a combination of DST-7 and DCT-8.
 12. A video signalprocessing apparatus for decoding a video signal, comprising: a memorystoring the video signal, and a decoder functionally coupled to thememory and configured to decoding-process the video signal, wherein thedecoder is configured to: generate a transform block including transformcoefficients by performing entropy decoding and inverse quantization onthe video signal, generate residual signals through an inverse transformfor the transform coefficients of the transform block based on a size ofthe transform block, and generate a reconstructed picture by generatinga prediction signal through prediction from the residual signals,wherein when generating the residual signals through the inversetransform for the transform coefficients of the transform block, thedecoder is configured to: not apply an inverse non-separable secondarytransform to the transform block when the size of the transform block issmaller than or equal to 4×4, apply the inverse non-separable secondarytransform to the transform block when the size of the transform block isgreater than 4×4, and apply an inverse primary transform to thetransform block separately in a horizontal direction and a verticaldirection.
 13. The video signal processing apparatus of claim 12,wherein the decoder is configured to apply the inverse non-separablesecondary transform to a left-top 4×4 region of the transform block whenthe size of the transform block is greater than 4×4.
 14. The videosignal processing apparatus of claim 10, wherein the inversenon-separable secondary transform includes at least one of a lowfrequency non-separable transform (LFNST) having a matrix of an L×16form, wherein the LFNST indicates a non-separable transform applied to aleft-top specific region and L is smaller than or equal to 16, a sparseorthonormal transform (SOT) having a matrix of a 16×16 form, a transformhaving 16 inputs and 16 outputs and based on Givens rotation, or alayered givens transform having 16 inputs and 16 outputs and applying adesignated permutation between two neighboring Givens rotation layers,wherein the forward primary transform includes a horizontal transformand a vertical transform for the transform block, and wherein each ofthe horizontal transform and the vertical transform is configured asDCT-2 or configured by a combination of DST-7 and DCT-8.